Mirroring a website where you need to log in first

Mirroring a website where you need to log in first

I needed to make an complete offline copy of a website the other day but was having difficulties as you needed to login to the site to get the content.  Luckily, an firefox add-on plus the wonderful wget came to the rescue.

Export cookies (https://addons.mozilla.org/en-US/firefox/addon/export-cookies/?src=api) is a firefox add-on that lets you export cookies in a format that wget understands.

So here is the procedure:
1) Log in to the website
2) Export the cookies into a cookies.txt file
3) Run wget
wget --load-cookies cookies.txt -mkEp  http://website.com
4) Hey-presto
https://addons.mozilla.org/en-US/firefox/addon/export-cookies/?src=api

Comments

Post a Comment

Popular posts from this blog

A census of amplified and overexpressed human cancer genes : Nature Reviews Cancer

This paper , that I am an author on, was released on 24th December 2009: "Integrated genome-wide screens of DNA copy number and gene expression in human cancers have accelerated the rate of discovery of amplified and overexpressed genes. However, the biological importance of most of the genes identified in such studies remains unclear. In this Analysis, we propose a weight-of-evidence based classification system for identifying individual genes in amplified regions that are selected for during tumour development. In a census of the published literature we have identified 77 genes for which there is good evidence of involvement in the development of human cancer." I spent a huge amount of time on this one but I am still pretty low on the author pecking order. This is for a super high impact journal, "Nature Reviews Cancer" which has a 2008 ISI Impact Factor of 30.762. Also check out the associated website which I put together just before Christmas. Hopefully, this...
Read more

RT @CancerInNorwich: 📢 Join us on Monday at 1pm for the next talk in our virtual seminar series. We will be hearing from Dr Wafa Al-Jamal from @QUBelfast, who will be talking about "Smart Nanomedicimes for Pancreatic Cancer". All are welcome to join via the MS Teams link: https://t.co/4xCYpjGuBZ https://t.co/EGyD4NAgeP

RT @CancerInNorwich: 📢 Join us on Monday at 1pm for the next talk in our virtual seminar series. We will be hearing from Dr Wafa Al-Jamal from @QUBelfast, who will be talking about "Smart Nanomedicimes for Pancreatic Cancer". All are welcome to join via the MS Teams link: https://t.co/4xCYpjGuBZ https://t.co/EGyD4NAgeP — Daniel Brewer (@danbrewer) May 11, 2023 from Twitter https://twitter.com/danbrewer
Read more

An excellent opportunity to join the cancer genetics team as a diagnostic lab bioinformatician in Norwich. Lots of exciting projects to get involved in. Deadline: 30/05/2023. Happy to chat about this role. https://t.co/SltGsxYF6Y https://t.co/zh08bqZFDb

An excellent opportunity to join the cancer genetics team as a diagnostic lab bioinformatician in Norwich. Lots of exciting projects to get involved in. Deadline: 30/05/2023. Happy to chat about this role. https://t.co/SltGsxYF6Y https://t.co/zh08bqZFDb — Daniel Brewer (@danbrewer) May 19, 2023 from Twitter https://twitter.com/danbrewer
Read more