Always Check for the Hidden API when Web Scraping

2 роки тому

# DISCORD (NEW): discord.gg/C4J2uckpbR
If this method if available, its the best way to scrape data from site. I will show you how to find the API endpoint that we can use to directly get the JSON data that is being sent from the server, before JavaScript gets its mucky paws on it and makes it look like what we see in our browser. Its quick and simple, and with just a few extra tips and techniques it can transform your web scraping.
# Scraper API: www.scrapingbee.com/?fpr=jhnwr
# Patreon: www.patreon.com/johnwatsonrooney
# Proxies: proxyscrape.com/?ref=jhnwr
# Hosting: Digital Ocean: m.do.co/c/c7c90f161ff6
# Gear I use: www.amazon.co.uk/shop/johnwatsonrooney

КОМЕНТАРІ: 525

@hectord.7107 Рік тому

I've doing this for years as a self taught programmer, there are some little tricks you did here that i didn't know, thank you for the video.

@JohnWatsonRooney Рік тому

Glad it was helpful!

@sworatex1683 Рік тому

It's my first year in programming and there was nothing new actually. I don't even think that pain was worth it I'd just make the scraper in js and make it return a json string.

@sworatex1683 Рік тому

But I guess that would be useless for bigger projects. I'd just do it in js if I want like an actual product list like this.

@gesuchter Рік тому

@D R lol

@mattshu Рік тому

@D R how do you “block” scraping??

@leow.2162 Рік тому

I'm not a very experience programmer, I've been doing it recreationally for like 2 years on and off but I did a lot of webscraping and this is just a really neat piece of knowledge that I wouldn't have come across own my own. Thanks.

@Davidca12 Рік тому

This single-handedly cut the running time of my program from literal hours to a couple of minutes, cannot thank you enough!

@JohnWatsonRooney Рік тому

Brilliant, thanks!

@zenon1903 2 роки тому

Please ignore my first comment. I checked out your first video in this series and learned about using scrapy shell to test each line of code. With that I found the bug in my code. The code worked PERFECTLY as advertised. Your the man! Much thanks!

@Kralnor Рік тому

This is a true gold nugget. Thanks for demonstrating how to easily view the request in Insomnia and auto-generate code!

@huisopperman 4 місяці тому

Thanks for sharing! This has helped me a lot. After struggling for weeks with selenium, I was able to apply this technique fairly quickly, and am now using it as source to scrape ETF-composition data to feed directly into a PowerBI dataset. Much appreciated!

@shivan2418 Рік тому

This is, no joke, the most useful video I ever saw on UKposts!

@abc.2924 2 роки тому

I've been using this trick for a while now, and I've learned it from you, so thanks. Amazing work man

@JohnWatsonRooney 2 роки тому

That’s great 👍

@fuad471 Рік тому

really nice and helpful tips in an actual topic with a sight-pleasuring recording quality, thank you for your time and efforts.

@krahkyn 4 місяці тому

This is such useful content that shows how much value experience gives - thank you for the straightforward and realistic tutorial!

@mujeebishaque 2 роки тому

I love you, John. You're awesome! Thanks for being unique and producing quality content.

@tikendraw 2 роки тому

I just want you to never stop creating such informative video. For god sake.

@user-fl8fy4wn1k Місяць тому

There is always something new to learn. I’ve been spending hours to grind such an information by hand-writing the whole program to get my result ;D Thanks!

@sajayagopi Рік тому

I was struggling with selinium to extract a table from javascript website. This video saved so much time. Thank you

@anduamlaktadesse9284 Рік тому

Hidden API is by far the easiest way to scrap a website!!!! Thanks bro!!! Big Clapping for you!!!!! I've followed all your procedures &finally i did it.

@wp4297 Рік тому

HUGE! I've been looking for this info for 2 days. 12 mins of your video better than anything else, by far. Thumbs up and thank you so much

@JohnWatsonRooney Рік тому

Thank you !!

@wp4297 Рік тому

@@JohnWatsonRooney you saved me a lot of time. I'm new to the topic, next days I'll take a look at you channel

@BIO5Error 2 роки тому

Very, very interesting - I'm going to give this a go myself. Cheers for another great video John.

@drkskwlkr Рік тому

Loved everything about this video! Great delivery style, production quality and interesting topic for me. First time visitor to this channel and not a Python user (thanks, UKposts, for your weird but helpful predictive algorithms).

@JohnWatsonRooney Рік тому

Thank you! I’m glad you enjoyed it

@ScottCov 2 роки тому

John Great Video...Thanks for taking the time to do this!!!

@sheikhakbar2067 2 роки тому

I always come to your channel for these excellent time-saving tips and tricks! Thank you!

@JohnWatsonRooney 2 роки тому

Glad you like them!

@klabauter-ev4ix 10 місяців тому

That was incredibly helpful and exactly what I needed today. Your presentation is very clear. Thank you!

@ERol-du3rd Рік тому

Awesome advice, a lot of people skip checking the requests when building scrapers but it can save a lot of time when it works

@ThijmenCodes Рік тому

Nice video! Used a similar method to collect European Court of Human Rights case documents since there is no official API. Glad to see such methods gaining popularity online, it’s so useful!

@GLo-qc8rz 2 місяці тому

OMG man, was searching for 3 hrs how to extract javascript data w/o complicated rendering and your vid gave a 3 second solution. thank you so much man

@angestellterderantifagmbh 7 місяців тому

My man, this is EXCACTLY what I was looking for. Had to do some extra steps, but a little try-and-error and basic understanding of HTTP was enough to solve my problem. Thank You!

@joshuakb2 Рік тому

This video came into my feed just a couple days after I used exactly this method to collect some data from a website. Very good info! This is much easier than web scraping. Unfortunately, in my case, the data I could get out of the API was incomplete, and each item in the response contained a URL to a page with the rest of the info I needed, so I had to write some code to fetch each of those pages and scrape the info I needed. But much easier than having to scrape the initial list, as well.

@JohnWatsonRooney Рік тому

Thanks! I’m glad it helped in some way. I often find that a combination of many different methods is needed to get the end result

@transatlant1c Рік тому

Nice video. It’s worth noting as well that many APIs will pageinate, so rather than checking how many total results exist and manually iterating over them - you just check to see if the ‘next page url’ or equivalent key exists in the results and if so, get that too until it doesn’t exist anymore, merging/appending each time until the dataset is complete 👍

@JohnWatsonRooney Рік тому

Yes you’re right thank you!

@ianroberts6531 Рік тому

In fact you can see at 05:33 that this particular API does just that - there's "nextPageURL" and "totalPages" at the end of the response JSON.

@mattimhiyasmith Рік тому

I have used the inspect with network method but wasn't aware of the copy as url method, thanks for that tip will save me a lot of time!

@gleysonoliveira802 2 роки тому

This video was the answer to my prayers! The next best option was to watch an one hour video and hope they would teach what you taught... In 10 minutes!!! 👏👏👏

@JohnWatsonRooney 2 роки тому

Thank you glad it helped!!

@Oiympus Рік тому

nice tips, it's always fun to poke around and look at what data webpages are using

@vinny723 2 роки тому

Great tutorial. My screen scrapping job went from 4.5 hours to 8 minutes!!!!!

@phoenixflower1225 Рік тому

Thank you so much - this is so insightful and educational. Really helped me understand so many things in so little time.

@freekdl6648 Рік тому

I rarely praise anything, but this tutorial was SO good! Well explained, no filler. In 7 or 8 minutes you guided me through finding the hidden information I needed, which tools I need to use and how to automate it. This tutorial gave me enough confidence to try to write my first Python script! Within hours I built a scraper that can pull all metadata for a full NFT collection from a marketplace. Without this video it would have taken days/weeks to discover all of this

@JohnWatsonRooney Рік тому

That awesome! thank you very kind!

@channul4887 Рік тому

"In 7 or 8 minutes" More like 11

@freekdl6648 Рік тому

@@channul4887 Nope! I had different goals so no need to follow the full tutorial

@MelonPython Рік тому

I even added it in my playlist. Great video. Definetely starting to love API's more and more.

@danielcardin9241 Рік тому

Because of this video, I was able to start my own rockets and satellites company. In only four hours, I started the company, launched thousands of rockets, and now I have my own interplanetary wireless intranet from which I can control the entire galaxy! Thanks again!

@Moiludi 5 місяців тому

Thank you! It provided a new way of thinking at the issue of collecting data. 🙏

@Josh-kw7zk 3 місяці тому

Thank you so much for this tutorial. It helped me a lot on my project. And i learn a lot of new things that i didnt know. Thank you!

@mrklopp1029 2 роки тому

Thank you for those videos. They're extremely helpful. Keep up the good work! 🙂

@JohnWatsonRooney 2 роки тому

Glad you like them!

@rameshks5281 2 роки тому

Easy to understand and very neat & clean narration. Keep it up 🙂

@JohnWatsonRooney 2 роки тому

Thanks a lot 😊

@muhammadrehan3030 2 роки тому

Thank you for such a wonderful videos. I learned a lot from you. BTW your video quality and background are always very beautiful.

@JohnWatsonRooney 2 роки тому

thanks! it's nice of you to mention video quality and background, i do my best!

@just_O Рік тому

Nice tutorial on scrapping, some tricks I have been using myself, and some others never heard of until now thx for sharing!!! Small adjustments if I may (please don't take this as criticism) I think you don't need to loop over each product to copy it to your res, you can use extend instead, also I think the header didn't change so you can take it out the loop over pages

@Jiloh5 Рік тому

It worked like charm! I really needed this. Thanks

@Jacob-gy9bg Рік тому

Wow, thanks for this excellent tutorial! I just spent all this time writing cumbersome Selenium code, when it turns out all the data I was looking for was already right there!

@JohnWatsonRooney Рік тому

Great! That’s exactly what I was hoping to achieve with this video

@brockobama257 Рік тому

bro, you're a game changer and i love you. if i ever see you in person ill offer to buy you a beer, or lunch, coffee whatever

@pascal831 7 місяців тому

Thanks John! You are a lifesaver sir!

@judgewooden Рік тому

I like how you regularly start sentences with 'you might think' assuming we are all idiots. I approve, glad smart people, like you, make time to explain to us plebs how the world works. Apprecated.

@JohnWatsonRooney Рік тому

Hey, thanks. I do my best to explain things how I would have wanted to be taught

@eakerz5642 Рік тому

Tnx :) Went from 1 hour scraping with Selenium to 1 minute just getting the JSONs.

@davida99 Рік тому

Wow! I just found a gem of a channel! Love your content!

@JohnWatsonRooney Рік тому

Thanks appreciated!!

@nadyamoscow2461 2 роки тому

Clear and helpful as usual. Thanks a lot!!

@davidl3383 Рік тому

brillant, i start to do that and it's very effective. Good chanel and good job. Thank you John

@unknownuser993 Рік тому

Wow that ‘generate code’ feature is super useful. Thanks!

@phoenixflower1225 Рік тому

This is seriously high level content right here

@lucasmoratoaraujo8433 10 місяців тому

Greetings from Brazil! Thank you! I just had to adjust some of the quote marks on the header (there were some 'chained' double quotes (like ""windows"")), making some of the header's strings be interpreted by python as code, not text. Just had to change inner double quotes for single quotes (e.g. "'windows'") and it worked perfectly!). Can't wait to try your other tutorials! Once more, thank your very much!

@JohnWatsonRooney 10 місяців тому

Hey! Thank you! I’m glad you managed to get it work

@irfanshaikh262 Рік тому

John, you make scraping interesting and motivating simultaneously. Good that I found your channel P. S. I lost myself it at 0:10😂

@ninjahkz4078 Рік тому

Lol, I hadn't thought of a possibility to get an api like that until now haha thanks a lot!

@inspecteurbane5666 Рік тому

Thanks a lot, very interesting video, i learned so many things that i didn't know. I will come back for sure!

@aidanomalley8607 Рік тому

Thank you, your videos has automated my job. All i need now is a AI cam of myself

@lokeshchowdary7487 Рік тому

Thank you for making this awesome tutorial.

@voinywolnyprod3046 Рік тому

Quite interesting! Thank you so much for showing such nice tricks, gonna get familiar with Insomnia.

@edgarasben Рік тому

This is amazing! So many things I didn’t know.

@tnssajivasudevan1601 2 роки тому

Great information Sir, really helpful.

@philippe6724 Рік тому

Super well taught! Thank you Sir!

@estring69 Рік тому

Thanks for the video. Not new to python, pandas, or API's, but do need to start scraping pages for the API as some are not published, or not documented well. Thanks.

@bigstupidtree3771 Рік тому

This has saved me hours today. Genuinely, thank you. 🙇‍♂️

@JohnWatsonRooney Рік тому

That’s great, thank you for watching!

@MARTIN-101 7 місяців тому

wonderful. thank you.

@cloudedcaesar 7 місяців тому

excellent video. Subscribed!

@basit_Ali01 8 місяців тому

This video is really amazing I learned web scraping from your videos thanks

@karthikshaindia 2 роки тому

Very nice... I did earlier the same on an another site, that was bit tricky. This is very straight forwarded site. Meanwhile , Insomnia reduced more works ;). Thanks for an another great video

@JohnWatsonRooney 2 роки тому

That’s great. Yes I picked this on because it was very easy, I think it helps people understand the core concepts better

@chadgray1745 2 роки тому

Thanks for this - and other - videos, John. Super helpful! Regarding the cookie expiring, can you suggest a way to use playwright to programmatically generate the cookie used on the API request? I am assuming that cookie isn’t the same as the cookie used for the request of the html but maybe that’s wrong?

@tubelessHuma 2 роки тому

This is very useful trick John. 💖

@JeanOfmArc Рік тому

You have shown me the light. Thank you for stopping me from making more web scripts that load up web pages in browsers to click buttons.

@JeanOfmArc Рік тому

I have tried this method, but sadly the site I am trying to scrape from returns "error": "invalid_client", "error_description": "The client credentials provided were invalid, the request is unauthorized." Am I out of luck?

@thecodegod2434 Рік тому

Thanks a lot John!!!

@Rob-ky1ob Рік тому

Instead of looping over the list and doing an append of each individual item, you can do list().extend(list()) which extends the list with the new list. The result of this is 1 list of dictionaries (basically an identical result to how you did it) but with less and cleaner code.

@vinubalank 2 роки тому

This has been a great help for a beginner like me.. Thanks a lot..

@MikeD-qx1kr 6 місяців тому

John, a specific video about how to scrape React website would be nice. It uses a mix of html and JSON data on pages...just an idea. Keep up the good work loving it.

@elliotnyberg9332 Рік тому

This is amazing and will help me alot. Thank you!!

@tobias2688 2 роки тому

Hi John, I loved the video so much that I had to join Patreon to subscribe there to you. Thanks!

@JohnWatsonRooney 2 роки тому

Hey! Thank you very much!

@michelramon5786 3 місяці тому

I was like "hm, okay, yeah" to "HOLY SHIT, THATS THE DOPEST SHIT I'VE EVER SEEN" I'm starting to get into this niche and I intend to learn more Python and SQL (you know, Data Analysis stuff/jobs) and I'm doing a project to scrape NBA statistics but there are always some errors and it ends up taking a long time. BUT THIS IS GOLD CONTENT, KEEP IT UP

@randyallen8610 Рік тому

Great content. Thanks for this video

@SkySesshomaru 2 роки тому

This is gold man, thank you! Just WOW.

@JohnWatsonRooney 2 роки тому

Thanks!

@rizkomao649 2 роки тому

Thank you, this is so inspiring.

@lagu1ful Рік тому

thank you for the information that you have explained, this is very helpful for the research I am doing

@lagu1ful Рік тому

thank you so much

@giovannimilana6428 Рік тому

Huge thanks this video was a game changer for me!

@ericbwertz Рік тому

Nice video -- perfect level of detail.

@JohnWatsonRooney Рік тому

Thanks

@bronxandbrenx 2 роки тому

My master in data extraction.

@BringMe_Back 2 роки тому

Amazing ♥️, super helpful

@abrammarba 3 місяці тому

Awesome! Thank you! 🙂

@TypicallyThomas Рік тому

Thanks so much. This makes things a lot easier

@JohnWatsonRooney Рік тому

Great to hear!

@TheEkkas Рік тому

Such a nice vid, if there was a VPN add, I didn't even notice it!

@HazelChikara Рік тому

You are the best sir!

@ahmedelsaid8368 6 місяців тому

amazing one ,thaks a lot

@im4485 2 роки тому

Hi John. Amazing content as always. Do you think I can skip learning scrappy for now? Can I do most of the scraping tasks just by using BS and request html?

@JohnWatsonRooney 2 роки тому

Sure you can. If it works for you then carry on!

@mrhide5690 Рік тому

Best! Thank you!

@codydabest Рік тому

This was nearly exactly my job back in 2014/2015 for a giant e-com shoe company. Was always nice when you'd come across a brand that included their inventory count in their API. But yes selenium/watir all day lol

@JohnWatsonRooney Рік тому

I’m often quite surprised how much info you can easily get!

@FireFly969 Рік тому

Amazing content and tricks

@user-eu6vy8sz4w 4 місяці тому

Great Stuff!

@amirahmed5905 Рік тому

Hi there, I found your channel where each and every video delicately made for web scrapping and automation which helps me a lot as work with web scraping and web automation. I have a request, if possible then please make python data post methods on Stateful api v1 and how to mimic cookies and session to get the job done. Thank you.

@JoeMSD Рік тому

Hello John! I have a quick question then - this will give you much more information then you might need if you don't change anything in Insomnia, will deleting the headers remove some of this? Or is it more likely to change based on what you put in the "for p in data " section? i.e. if I include keys that are further indented, will I just receive those keys as oppose to everything nested in an undented key? I found this extremely useful though and have been using it since!

@agsantiago22 2 роки тому

Great video! Thanks so much for sharing! I think you should consider some academic research program (if you havent't already). I am sure you would do an amazing work. Congrats and thanks again!

@JohnWatsonRooney 2 роки тому

Thanks for watching I’m glad you enjoyed it!