off topic -- web testing/scraping question.

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hi guys.

Dealing with an issue -- waaaaaay off topic fr fed! (but thought I' post, see if anyone has thoughts..)

I'm dealing with a testing/scraping process of a target site. Collecting isbn data for college classes. The site has gone to using obfuscation/encryption/etc.. which requires implementing a browser/_javascript_ soln to generate the actual content.

The 1st pass test uses headless browser/casperjs to simply get the target page. This solution works, but is abysmally slow. In fact I can manually insert the url into a browser and get the returned result faster!

I've seen some articles that imply it's doable to fire off/run a real browser ff/chrome from the cmd line with the targetd url ,which would then produce the required output. (But haven't seen any pointers/exmples on how to accomplish as of yet).

Any thoughts/comments/pointers??

Thanks..

ps/ I'll eventually post to SO (stackoverflow), and i've got some ongoing initial conversations on a few IRC channels. If you can think of other places I could check, I'm even thinking of finding a resource that might be able to "reverse" engineer the obfuscated content (for $$$) if I knew a good site/resource to approach.

_______________________________________________
users mailing list -- users@xxxxxxxxxxxxxxxxxxxxxxx
To unsubscribe send an email to users-leave@xxxxxxxxxxxxxxxxxxxxxxx
[Index of Archives]     [Older Fedora Users]     [Fedora Announce]     [Fedora Package Announce]     [EPEL Announce]     [EPEL Devel]     [Fedora Magazine]     [Fedora Summer Coding]     [Fedora Laptop]     [Fedora Cloud]     [Fedora Advisory Board]     [Fedora Education]     [Fedora Security]     [Fedora Scitech]     [Fedora Robotics]     [Fedora Infrastructure]     [Fedora Websites]     [Anaconda Devel]     [Fedora Devel Java]     [Fedora Desktop]     [Fedora Fonts]     [Fedora Marketing]     [Fedora Management Tools]     [Fedora Mentors]     [Fedora Package Review]     [Fedora R Devel]     [Fedora PHP Devel]     [Kickstart]     [Fedora Music]     [Fedora Packaging]     [Fedora SELinux]     [Fedora Legal]     [Fedora Kernel]     [Fedora OCaml]     [Coolkey]     [Virtualization Tools]     [ET Management Tools]     [Yum Users]     [Yosemite News]     [Gnome Users]     [KDE Users]     [Fedora Art]     [Fedora Docs]     [Fedora Sparc]     [Libvirt Users]     [Fedora ARM]
  Powered by Linux