crawlergo是一个使用chrome headless模式进行URL收集的浏览器爬虫。它对整个网页的关键位置与DOM渲染阶段进行HOOK,自动进行表单填充并提交,配合智能的JS事件触发,尽可能的收集网站暴露出的入口。内置URL去重模块,过滤掉了大量伪静态URL,对于大型网站仍保持较快的解析与抓取速度,最后得到高质量的请求结果集合。
crawlergo 目前支持以下特性:
-
原生浏览器环境,协程池调度任务
-
表单智能填充、自动化提交
-
完整DOM事件收集,自动化触发
-
智能URL去重,去掉大部分的重复请求
-
全面分析收集,包括javascript文件内容、页面注释、robots.txt文件和常见路径Fuzz
-
支持Host绑定,自动添加Referer
-
支持请求代理,支持爬虫结果主动推送
Github: https://github.com/Qianlitp/crawlergo
作者开源了源码,我是很兴奋的,以前也有写一个的想法,但是开源的动态爬虫不多,看了其中几个。
调研
1.https://github.com/fcavallarin/htcap
•递归dom搜索引擎•发现ajax/fetch/jsonp/websocket请求•支持cookie,代理,ua,http auth•基于文本相似度的页面重复数据删除引擎
•根据文本长度 <256
•simhash
•else
•ShinglePrint
•主要代码是python调用puppeteer,但是核心逻辑在js里
2.https://github.com/go-rod/rod
•一个操作chrome headless的go库•它比官方提供的chrome操作库更容易使用•有效解决了chrome残留僵尸进程的问题
3.https://github.com/lc/gau
•通过一些通用接口获取url信息
4.https://github.com/jaeles-project/gospider•Web静态爬虫,也提供了一些方法获取更多URL
5.https://github.com/chaitin/rad1.rad虽然没有开源,但是它里面使用yaml进行的配置选项很多,通过配置选项可以大致知道它的一些特性。2.可以手动登陆3.启用图片4.显示对爬取url的一些限制
1.不允许的文件后缀2.不允许的url关键字3.不允许的域名4.不允许的url
5.设置下个页面最大点击和事件触发Crawlergo
之前也想过写一个动态爬虫来对接扫描器,但是动态爬虫有很多细节都需要打磨,一直没时间做,现在有现成的源码参考能省下不少事。
主要看几个点
•对浏览器 JavaScript环境的hoook
•dom的触发,表单填充
•url如何去重•url的收集
目录结构
├─cmd│ └─crawlergo# 程序主入口├─examples├─imgs└─pkg├─config# 一些配置相关├─engine# chrome相关程序├─filter# 去重相关├─js# 一些注入的js├─logger# 日志├─model# url和请求相关的库└─tools# 一些通用类库└─requests
根据源码的调用堆栈做了一个程序启动流程图
配置文件
pkg/config/config.go定义了一些默认的配置文件
const(DefaultUA="Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.0 Safari/537.36"MaxTabsCount=10TabRunTimeout=20 time.SecondDefaultInputText ="Crawlergo"// 默认输入的文字FormInputKeyword ="Crawlergo"// form输入的文字,但是代码中没有引用这个变量的SuspectURLRegex = (?:"|')(((?:[a-zA-Z]{1,10}://|//)[^"'/]{1,}\.[a-zA-Z]{2,}[^"']{0,})|((?:/|\.\./|\./)[^"'><,;|*()(%%$^/\\\[\]][^"'><,;|()]{1,})|([a-zA-Z0-9_\-/]{1,}/[a-zA-Z0-9_\-/]{1,}\.(?:[a-zA-Z]{1,4}|action)(?:[\?|#][^"|']{0,}|))|([a-zA-Z0-9_\-/]{1,}/[a-zA-Z0-9_\-/]{3,}(?:[\?|#][^"|']{0,}|))|([a-zA-Z0-9_\-]{1,}\.(?:php|asp|aspx|jsp|json|action|html|js|txt|xml)(?:[\?|#][^"|']{0,}|)))(?:"|')//url获取正则URLRegex= ((https?|ftp|file):)?//[-A-Za-z0-9+&@#/%?=~_|!:,.;]+[-A-Za-z0-9+&@#/%=~_|] // url获取正则AttrURLRegex = ``DomContentLoadedTimeout=5 time.SecondEventTriggerInterval =100 time.Millisecond// 单位毫秒BeforeExitDelay=1 time.SecondDefaultEventTriggerMode= EventTriggerAsyncMaxCrawlCount=200)// 请求的来源,记录了每个url的来源,可以根据这些关键词定位到相关的获取代码const(FromTarget ="Target"//初始输入的目标FromNavigation ="Navigation"//页面导航请求FromXHR ="XHR"//ajax异步请求FromDOM ="DOM"//dom解析出来的请求FromJSFile ="JavaScript"//JS脚本中解析FromFuzz ="PathFuzz"//初始path fuzzFromRobots ="robots.txt"//robots.txtFromComment ="Comment"//页面中的注释FromWebSocket="WebSocket"FromEventSource="EventSource"FromFetch="Fetch"FromHistoryAPI="HistoryAPI"FromOpenWindow="OpenWindow"FromHashChange="HashChange"FromStaticRes="StaticResource"FromStaticRegex="StaticRegex")// 静态文件后缀,这些后缀的文件全部阻断读取varStaticSuffix = []string{"png","gif","jpg","mp4","mp3","mng","pct","bmp","jpeg","pst","psp","ttf","tif","tiff","ai","drw","wma","ogg","wav","ra","aac","mid","au","aiff","dxf","eps","ps","svg","3gp","asf","asx","avi","mov","mpg","qt","rm","wmv","m4a","bin","xls","xlsx","ppt","pptx","doc","docx","odt","ods","odg","odp","exe","zip","rar","tar","gz","iso","rss","pdf","txt","dll","ico","gz2","apk","crt","woff","map","woff2","webp","less","dmg","bz2","otf","swf","flv","mpeg","dat","xsl","csv","cab","exif","wps","m4v","rmvb",}// 动态文件的后缀,过滤器用于过滤伪静态用varScriptSuffix = []string{"php","asp","jsp","asa",}// 默认不爬的url关键词varDefaultIgnoreKeywords = []string{"logout","quit","exit"}// 填充表单相关,这些是模糊匹配,匹配到了则使用对应的规则varAllowedFormName = []string{"default","mail","code","phone","username","password","qq","id_card","url","date","number"}varInputTextMap = map[string]map[string]interface{}{"mail": {"keyword": []string{"mail"},"value":"crawlergo@gmail.com",},"code": {"keyword": []string{"yanzhengma","code","ver","captcha"},"value":"123a",},"phone": {"keyword": []string{"phone","number","tel","shouji"},"value":"18812345678",},"username": {"keyword": []string{"name","user","id","login","account"},"value":"crawlergo@gmail.com",},"password": {"keyword": []string{"pass","pwd"},"value":"Crawlergo6.",},"qq": {"keyword": []string{"qq","wechat","tencent","weixin"},"value":"123456789",},"IDCard": {"keyword": []string{"card","shenfen"},"value":"511702197409284963",},"url": {"keyword": []string{"url","site","web","blog","link"},"value":"https://crawlergo.nice.cn/",},"date": {"keyword": []string{"date","time","year","now"},"value":"2018-01-01",},"number": {"keyword": []string{"day","age","num","count"},"value":"10",},}
URL过滤方式
crawlergo有两种过滤,fensimple和smart。
simple
simple过滤方式比较简单,就是将计算请求体的method、url、postdata 结合计算md5
判断是否存在
smart
smart过滤会对每个请求的参数name,参数value,path 进行标记,会标记成以下字段
const(CustomValueMark="{{Crawlergo}}"FixParamRepeatMark="{{fix_param}}"FixPathMark="{{fix_path}}"TooLongMark="{{long}}"NumberMark="{{number}}"ChineseMark="{{chinese}}"UpperMark="{{upper}}"LowerMark="{{lower}}"UrlEncodeMark="{{urlencode}}"UnicodeMark="{{unicode}}"BoolMark="{{bool}}"ListMark="{{list}}"TimeMark="{{time}}"MixAlphaNumMark="{{mix_alpha_num}}"MixSymbolMark="{{mix_symbol}}"MixNumMark="{{mix_num}}"NoLowerAlphaMark="{{no_lower}}"MixStringMark="{{mix_str}}")
例如
?m=home&c=index&a=index?type=202cb962ac59075b964b07152d234b70?id=1?msg=%E6%B6%88%E6%81%AF
处理结果即
?m=home&c=index&a=index?type={hash}?id={int}?msg={urlencode}
会对超过阈值的结果,以及计算的唯一id进行比对,来判断是否过滤,总体来说是基于url的过滤方式。
使用的正则
varchineseRegex = regexp.MustCompile("[\u4e00-\u9fa5]+")varurlencodeRegex = regexp.MustCompile("(?:%[A-Fa-f0-9]{2,6})+")varunicodeRegex = regexp.MustCompile((?:\\u\w{4})+)varonlyAlphaRegex = regexp.MustCompile("^[a-zA-Z]+$")varonlyAlphaUpperRegex = regexp.MustCompile("^[A-Z]+$")varalphaUpperRegex = regexp.MustCompile("[A-Z]+")varalphaLowerRegex = regexp.MustCompile("[a-z]+")varreplaceNumRegex = regexp.MustCompile([0-9]+\.[0-9]+|\d+)varonlyNumberRegex = regexp.MustCompile(^[0-9]+$)varnumberRegex = regexp.MustCompile([0-9]+)varOneNumberRegex = regexp.MustCompile([0-9])varnumSymbolRegex = regexp.MustCompile(\.|_|-)vartimeSymbolRegex = regexp.MustCompile(-|:|\s)varonlyAlphaNumRegex = regexp.MustCompile(^[0-9a-zA-Z]+$)varmarkedStringRegex = regexp.MustCompile(^{{.+}}$)varhtmlReplaceRegex = regexp.MustCompile(\.shtml|\.html|\.htm)
对路径的处理
/标记路径/func (s SmartFilter) MarkPath(pathstring)string{pathParts:= strings.Split(path,"/")forindex, part := range pathParts {iflen(part) >=32{pathParts[index] = TooLongMark}elseifonlyNumberRegex.MatchString(numSymbolRegex.ReplaceAllString(part,"")) {pathParts[index] = NumberMark}elseifstrings.HasSuffix(part,".html") || strings.HasSuffix(part,".htm") || strings.HasSuffix(part,".shtml") {part= htmlReplaceRegex.ReplaceAllString(part,"")// 大写、小写、数字混合ifnumberRegex.MatchString(part) &&alphaUpperRegex.MatchString(part) &&alphaLowerRegex.MatchString(part) {pathParts[index] = MixAlphaNumMark// 纯数字}elseifb := numSymbolRegex.ReplaceAllString(part,""); onlyNumberRegex.MatchString(b) {pathParts[index] = NumberMark}// 含有特殊符号}elseifs.hasSpecialSymbol(part) {pathParts[index] = MixSymbolMark}elseifchineseRegex.MatchString(part) {pathParts[index] = ChineseMark}elseifunicodeRegex.MatchString(part) {pathParts[index] = UnicodeMark}elseifonlyAlphaUpperRegex.MatchString(part) {pathParts[index] = UpperMark// 均为数字和一些符号组成}elseifb := numSymbolRegex.ReplaceAllString(part,""); onlyNumberRegex.MatchString(b) {pathParts[index] = NumberMark// 数字出现的次数超过3,视为伪静态path}elseifb := OneNumberRegex.ReplaceAllString(part,"0"); strings.Count(b,"0") > 3 {pathParts[index] = MixNumMark}}newPath:= strings.Join(pathParts,"/")returnnewPath}
基于网页结构去重
作者原帖中的基于网页结构去重写的非常精彩 https://www.anquanke.com/post/id/178339#h2-17
参考的论文下载: https://patents.google.com/patent/CN101694668B/zh
作者的将网页特征向量抽离出来,索引存储,用来判断大量网页的相似度,非常惊艳,未来用到资产收集系统或者网络空间引擎上也都是非常不错的选择。
URL收集
robots
在爬虫之前,会先请求robots.txt,解析出所有链接,加入到待爬取页面。
源码中使用了一个正则来匹配
varurlFindRegex = regexp.MustCompile((?:Disallow|Allow):.*?(/.+))
看了下robots规范:https://baike.baidu.com/item/robots%E5%8D%8F%E8%AE%AE/2483797 ,应该还可以再优化一下,来处理一些表达式。
DIR FUZZ
在爬虫之前,如果没有指定dir字典的话,默认会使用内置的字典
['11', '123', '2017', '2018', 'message', 'mis', 'model', 'abstract', 'account', 'act', 'action', 'activity', 'ad', 'address', 'ajax', 'alarm', 'api', 'app', 'ar', 'attachment', 'auth', 'authority', 'award', 'back', 'backup', 'bak', 'base', 'bbs', 'bbs1', 'cms', 'bd', 'gallery', 'game', 'gift', 'gold', 'bg', 'bin', 'blacklist', 'blog', 'bootstrap', 'brand', 'build', 'cache', 'caches', 'caching', 'cacti', 'cake', 'captcha', 'category', 'cdn', 'ch', 'check', 'city', 'class', 'classes', 'classic', 'client', 'cluster', 'collection', 'comment', 'commit', 'common', 'commons', 'components', 'conf', 'config', 'mysite', 'confs', 'console', 'consumer', 'content', 'control', 'controllers', 'core', 'crontab', 'crud', 'css', 'daily', 'dashboard', 'data', 'database', 'db', 'default', 'demo', 'dev', 'doc', 'download', 'duty', 'es', 'eva', 'examples', 'excel', 'export', 'ext', 'fe', 'feature', 'file', 'files', 'finance', 'flashchart', 'follow', 'forum', 'frame', 'framework', 'ft', 'group', 'gss', 'hello', 'helper', 'helpers', 'history', 'home', 'hr', 'htdocs', 'html', 'hunter', 'image', 'img11', 'import', 'improve', 'inc', 'include', 'includes', 'index', 'info', 'install', 'interface', 'item', 'jobconsume', 'jobs', 'json', 'kindeditor', 'l', 'languages', 'lib', 'libraries', 'libs', 'link', 'lite', 'local', 'log', 'login', 'logs', 'mail', 'main', 'maintenance', 'manage', 'manager', 'manufacturer', 'menus', 'models', 'modules', 'monitor', 'movie', 'mysql', 'n', 'nav', 'network', 'news', 'notice', 'nw', 'oauth', 'other', 'page', 'pages', 'passport', 'pay', 'pcheck', 'people', 'person', 'php', 'phprpc', 'phptest', 'picture', 'pl', 'platform', 'pm', 'portal', 'post', 'product', 'project', 'protected', 'proxy', 'ps', 'public', 'qq', 'question', 'quote', 'redirect', 'redisclient', 'report', 'resource', 'resources', 's', 'save', 'schedule', 'schema', 'script', 'scripts', 'search', 'security', 'server', 'service', 'shell', 'show', 'simple', 'site', 'sites', 'skin', 'sms', 'soap', 'sola', 'sort', 'spider', 'sql', 'stat', 'static', 'statistics', 'stats', 'submit', 'subways', 'survey', 'sv', 'syslog', 'system', 'tag', 'task', 'tasks', 'tcpdf', 'template', 'templates', 'test', 'tests', 'ticket', 'tmp', 'token', 'tool', 'tools', 'top', 'tpl', 'txt', 'upload', 'uploadify', 'uploads', 'url', 'user', 'util', 'v1', 'v2', 'vendor', 'view', 'views', 'web', 'weixin', 'widgets', 'wm', 'wordpress', 'workspace', 'ws', 'www', 'www2', 'wwwroot', 'zone', 'admin', 'admin_bak', 'mobile', 'm', 'js']
根据状态码判断
爬虫时的url收集
•解析流量中的url
•获取xmr类型的请求
•正则解析js、html、json中url•收集当前页面上的url信息
"src","href","data-url","data-href"object[data]注释中的url
对浏览器环境的hook
在chrome的tab初始化时,会执行一段js代码,hook部分函数和事件来控制js的运行环境。
同时定义了一个js全局函数addLink、Test,通过这个函数可以与go进行交互。
调用go函数
js初始化时对这个函数重新包装
constbinding =window["addLink"];window["addLink"] =async(...args) => {constme =window["addLink"];letcallbacks = me['callbacks'];if(!callbacks) {callbacks =newMap();me['callbacks'] = callbacks;}constseq = (me['lastSeq'] ||0) +1;me['lastSeq'] = seq;constpromise =newPromise(fulfill => callbacks.set(seq, fulfill));binding(JSON.stringify({name:"addLink", seq, args}));returnpromise;};constbindingTest =window["Test"];window["Test"] =async(...args) => {constme =window["Test"];letcallbacks = me['callbacks'];if(!callbacks) {callbacks =newMap();me['callbacks'] = callbacks;}constseq = (me['lastSeq'] ||0) +1;me['lastSeq'] = seq;constpromise =newPromise(fulfill => callbacks.set(seq, fulfill));binding(JSON.stringify({name:"Test", seq, args}));returnpromise;};
go处理逻辑
执行完go函数后会再执行一段js
constDeliverResultJS = `(functiondeliverResult(name, seq, result){window[name]['callbacks'].get(seq)(result);window[name]['callbacks'].delete(seq);})("%s", %v,"%s")
但是没看懂使用promise后回调调用的意义是什么。。
Bypass headless detect
// Pass the Webdriver Test.Object.defineProperty(navigator,'webdriver', {get:()=>false,});// Pass the Plugins Length Test.// Overwrite the plugins property to use a custom getter.Object.defineProperty(navigator,'plugins', {// This just needs to have length > 0 for the current test,// but we could mock the plugins too if necessary.get:()=>[1,2,3,4,5],});// Pass the Chrome Test.// We can mock thisinasmuch depthaswe needforthe test.window.chrome = {runtime: {},};// Pass the Permissions Test.constoriginalQuery =window.navigator.permissions.query;window.navigator.permissions.query =(parameters)=>(parameters.name ==='notifications'?Promise.resolve({state: Notification.permission }) :originalQuery(parameters));//Pass the Permissions Test. navigator.userAgentObject.defineProperty(navigator,'userAgent', {get:()=>"Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.0 Safari/537.36",});// 修改浏览器对象的属性Object.defineProperty(navigator,'platform', {get:function(){return'win32'; }});Object.defineProperty(navigator,'language', {get:function(){return'zh-CN'; }});Object.defineProperty(navigator,'languages', {get:function(){return["zh-CN","zh"]; }});
记录url以及对前端框架的适配
//historyapi hook 许多前端框架都采用此API进行页面路由,记录url并取消操作window.history.pushState =function(a, b, c){window.addLink(c,"HistoryAPI");}window.history.replaceState =function(a, b, c){window.addLink(c,"HistoryAPI");}Object.defineProperty(window.history,"pushState",{"writable":false,"configurable":false});Object.defineProperty(window.history,"replaceState",{"writable":false,"configurable":false});// 监听hash改变 Vue等框架默认使用hash部分进行前端页面路由window.addEventListener("hashchange",function(){window.addLink(document.location.href,"HashChange");});// 监听窗口的打开和关闭,记录新窗口打开的url,并取消实际操作//hookwindow.openwindow.open =function(url){console.log("trying to open window.");window.addLink(url,"OpenWindow");}Object.defineProperty(window,"open",{"writable":false,"configurable":false});//hookwindowclosewindow.close =function(){console.log("trying to close page.");};Object.defineProperty(window,"close",{"writable":false,"configurable":false});//hookwindow.WebSocket 、window.EventSource 、window.fetch 等函数varoldWebSocket =window.WebSocket;window.WebSocket =function(url, arg){window.addLink(url,"WebSocket");returnnewoldWebSocket(url, arg);}varoldEventSource =window.EventSource;window.EventSource =function(url){window.addLink(url,"EventSource");returnnewoldEventSource(url);}varoldFetch =window.fetch;window.fetch =function(url){window.addLink(url,"Fetch");returnoldFetch(url);}
hook setTimeout/SetInterval
// hooksetTimeout//window.__originalSetTimeout =window.setTimeout;//window.setTimeout = function() {// arguments[1] = 0;//returnwindow.__originalSetTimeout.apply(this, arguments);//};//Object.defineProperty(window,"setTimeout",{"writable":false,"configurable":false});// hooksetInterval 时间设置为60秒 目的是减轻chrome的压力window.__originalSetInterval =window.setInterval;window.setInterval =function(){arguments[1] =60000;returnwindow.__originalSetInterval.apply(this,arguments);};Object.defineProperty(window,"setInterval",{"writable":false,"configurable":false});
这个hook操作没有明白,将setInterval强制设为了60s,我想应该来个判断,大于60s时再统一设置,60s,这样爬虫效率就变得太低了。
setTimeout取消了hook,毕竟只执行一次,可能不是很重要。
hook ajax
hook 原生ajax并限制最大请求数,可能是怕自动点击造成ajax爆炸
// 劫持原生ajax,并对每个请求设置最大请求次数window.ajax_req_count_sec_auto = {};XMLHttpRequest.prototype.__originalOpen = XMLHttpRequest.prototype.open;XMLHttpRequest.prototype.open =function(method, url, async, user, password){// hook codethis.url = url;this.method = method;letname = method + url;if(!window.ajax_req_count_sec_auto.hasOwnProperty(name)) {window.ajax_req_count_sec_auto[name] =1}else{window.ajax_req_count_sec_auto[name] +=1}if(window.ajax_req_count_sec_auto[name] <=10) {returnthis.__originalOpen(method, url,true, user, password);}}Object.defineProperty(XMLHttpRequest.prototype,"open",{"writable":false,"configurable":false});XMLHttpRequest.prototype.__originalSend = XMLHttpRequest.prototype.send;XMLHttpRequest.prototype.send =function(data){// hook codeletname =this.method +this.url;if(window.ajax_req_count_sec_auto[name] <=10) {returnthis.__originalSend(data);}}Object.defineProperty(XMLHttpRequest.prototype,"send",{"writable":false,"configurable":false});XMLHttpRequest.prototype.__originalAbort = XMLHttpRequest.prototype.abort;XMLHttpRequest.prototype.abort =function(){// hook code}Object.defineProperty(XMLHttpRequest.prototype,"abort",{"writable":false,"configurable":false});
锁定表单重置
爬虫在处理网页时,会先填充表单,接着触发事件去提交表单,但有时会意外点击到表单的重置按钮,造成内容清空,表单提交失败。所以为了防止这种情况的发生,需要Hook表单的重置并锁定不能修改。
// 锁定表单重置HTMLFormElement.prototype.reset =function(){console.log("cancel reset form")};Object.defineProperty(HTMLFormElement.prototype,"reset",{"writable":false,"configurable":false});
事件Hook
对dom0级和dom2级事件分别hook,用一个数组设定最大触发次数,所有js调用的事件,会对每个标签设置一个sec_auto_dom2_event_flag属性,方便后面寻找并自动触发。
// hook dom2 级事件监听window.add_even_listener_count_sec_auto = {};// record event func , hook addEventListenerletold_event_handle = Element.prototype.addEventListener;Element.prototype.addEventListener =function(event_name, event_func, useCapture){letname ="<"+this.tagName +"> "+this.id +this.name +this.getAttribute("class") +"|"+ event_name;//console.log(name)// 对每个事件设定最大的添加次数,防止无限触发,最大次数为5if(!window.add_even_listener_count_sec_auto.hasOwnProperty(name)) {window.add_even_listener_count_sec_auto[name] =1;}elseif(window.add_even_listener_count_sec_auto[name] ==5) {return;}else{window.add_even_listener_count_sec_auto[name] +=1;}if(this.hasAttribute("sec_auto_dom2_event_flag")) {letsec_auto_dom2_event_flag =this.getAttribute("sec_auto_dom2_event_flag");this.setAttribute("sec_auto_dom2_event_flag", sec_auto_dom2_event_flag +"|"+ event_name);}else{this.setAttribute("sec_auto_dom2_event_flag", event_name);}old_event_handle.apply(this,arguments);};functiondom0_listener_hook(that, event_name){letname ="<"+ that.tagName +"> "+ that.id + that.name + that.getAttribute("class") +"|"+ event_name;//console.log(name);// 对每个事件设定最大的添加次数,防止无限触发,最大次数为5if(!window.add_even_listener_count_sec_auto.hasOwnProperty(name)) {window.add_even_listener_count_sec_auto[name] =1;}elseif(window.add_even_listener_count_sec_auto[name] ==5) {return;}else{window.add_even_listener_count_sec_auto[name] +=1;}if(that.hasAttribute("sec_auto_dom2_event_flag")) {letsec_auto_dom2_event_flag = that.getAttribute("sec_auto_dom2_event_flag");that.setAttribute("sec_auto_dom2_event_flag", sec_auto_dom2_event_flag +"|"+ event_name);}else{that.setAttribute("sec_auto_dom2_event_flag", event_name);}}// hook dom0 级事件监听Object.defineProperties(HTMLElement.prototype, {onclick: {set:function(newValue){onclick = newValue;dom0_listener_hook(this,"click");}},onchange: {set:function(newValue){onchange = newValue;dom0_listener_hook(this,"change");}},onblur: {set:function(newValue){onblur = newValue;dom0_listener_hook(this,"blur");}},ondblclick: {set:function(newValue){ondblclick = newValue;dom0_listener_hook(this,"dbclick");}},onfocus: {set:function(newValue){onfocus = newValue;dom0_listener_hook(this,"focus");}},onkeydown: {set:function(newValue){onkeydown = newValue;dom0_listener_hook(this,"keydown");}},onkeypress: {set:function(newValue){onkeypress = newValue;dom0_listener_hook(this,"keypress");}},onkeyup: {set:function(newValue){onkeyup = newValue;dom0_listener_hook(this,"keyup");}},onload: {set:function(newValue){onload = newValue;dom0_listener_hook(this,"load");}},onmousedown: {set:function(newValue){onmousedown = newValue;dom0_listener_hook(this,"mousedown");}},onmousemove: {set:function(newValue){onmousemove = newValue;dom0_listener_hook(this,"mousemove");}},onmouseout: {set:function(newValue){onmouseout = newValue;dom0_listener_hook(this,"mouseout");}},onmouseover: {set:function(newValue){onmouseover = newValue;dom0_listener_hook(this,"mouseover");}},onmouseup: {set:function(newValue){onmouseup = newValue;dom0_listener_hook(this,"mouseup");}},onreset: {set:function(newValue){onreset = newValue;dom0_listener_hook(this,"reset");}},onresize: {set:function(newValue){onresize = newValue;dom0_listener_hook(this,"resize");}},onselect: {set:function(newValue){onselect = newValue;dom0_listener_hook(this,"select");}},onsubmit: {set:function(newValue){onsubmit = newValue;dom0_listener_hook(this,"submit");}},onunload: {set:function(newValue){onunload = newValue;dom0_listener_hook(this,"unload");}},onabort: {set:function(newValue){onabort = newValue;dom0_listener_hook(this,"abort");}},onerror: {set:function(newValue){onerror = newValue;dom0_listener_hook(this,"error");}},})
表单填充,事件触发
表单填充
input处理
// 找出type为空 或者type=textfor_, node := range nodes {// 兜底超时tCtxN, cancelN := context.WithTimeout(ctx,time.Second*5)attrType:= node.AttributeValue("type")ifattrType =="text"|| attrType ==""{inputName:= node.AttributeValue("id") + node.AttributeValue("class") + node.AttributeValue("name")value:= f.GetMatchInputText(inputName)// 寻找匹配类型的值varnodeIds = []cdp.NodeID{node.NodeID}// 先使用模拟输入_ = chromedp.SendKeys(nodeIds,value, chromedp.ByNodeID).Do(tCtxN)// 再直接赋值JS属性_ = chromedp.SetAttributeValue(nodeIds,"value",value, chromedp.ByNodeID).Do(tCtxN)}elseifattrType =="email"|| attrType =="password"|| attrType =="tel"{value:= f.GetMatchInputText(attrType)// 寻找匹配类型的值varnodeIds = []cdp.NodeID{node.NodeID}// 先使用模拟输入_ = chromedp.SendKeys(nodeIds,value, chromedp.ByNodeID).Do(tCtxN)// 再直接赋值JS属性_ = chromedp.SetAttributeValue(nodeIds,"value",value, chromedp.ByNodeID).Do(tCtxN)}elseifattrType =="radio"|| attrType =="checkbox"{varnodeIds = []cdp.NodeID{node.NodeID}= chromedp.SetAttributeValue(nodeIds,"checked","true", chromedp.ByNodeID).Do(tCtxN)}elseifattrType =="file"|| attrType =="image"{varnodeIds = []cdp.NodeID{node.NodeID}wd, _ := os.Getwd()filePath:= wd +"/upload/image.png"= chromedp.RemoveAttribute(nodeIds,"accept", chromedp.ByNodeID).Do(tCtxN)= chromedp.RemoveAttribute(nodeIds,"required", chromedp.ByNodeID).Do(tCtxN)// 对于一些简单的限制,可以去掉,比如找到文件上传的dom节点并删除 accept 和 required 属性:= chromedp.SendKeys(nodeIds, filePath, chromedp.ByNodeID).Do(tCtxN)}cancelN()}
multiSelect
css语法获取select第一个元素,设置属性即可
optionNodes, optionErr := f.tab.GetNodeIDs(selectoption:first-child)ifoptionErr !=nil|| len(optionNodes) ==0{logger.Logger.Debug("fillMultiSelect: getselectoptionelementerr")ifoptionErr !=nil{logger.Logger.Debug(optionErr)}return}= chromedp.SetAttributeValue(optionNodes,"selected","true", chromedp.ByNodeID).Do(tCtx)= chromedp.SetJavascriptAttribute(optionNodes,"selected","true", chromedp.ByNodeID).Do(tCtx)
TextArea
找到后填充即可
textareaNodes, textareaErr := f.tab.GetNodeIDs(textarea)iftextareaErr !=nil|| len(textareaNodes) ==0{logger.Logger.Debug("fillTextarea:gettextareaelementerr")iftextareaErr !=nil{logger.Logger.Debug(textareaErr)}return}_= chromedp.SendKeys(textareaNodes, value, chromedp.ByNodeID).Do(tCtx)
自动化提交表单
提交表单也有一些需要注意的问题,直接点击form表单的提交按钮会导致页面重载,我们并不希望当前页面刷新,所以除了Hook住前端导航请求之外,我们还可以为form节点设置target属性,指向一个隐藏的iframe。具体操作的话就是新建隐藏iframe然后将form表单的target指向它即可
/设置form的target指向一个frame/constNewFrameTemplate = (functionsec_auto_new_iframe(){letframe =document.createElement("iframe");frame.setAttribute("name","%s");frame.setAttribute("id","%s");frame.setAttribute("style","display: none");document.body.appendChild(frame);})()func(tab Tab) setFormToFrame() {// 首先新建 framenameStr := tools.RandSeq(8)tab.Evaluate(fmt.Sprintf(js.NewFrameTemplate,nameStr,nameStr))// 接下来将所有的 form 节点target都指向它ctx:= tab.GetExecutor()formNodes, formErr := tab.GetNodeIDs(form)ifformErr !=nil|| len(formNodes) ==0{logger.Logger.Debug("setFormToFrame:getformelementerr")ifformErr !=nil{logger.Logger.Debug(formErr)}return}tCtx, cancel := context.WithTimeout(ctx,time.Second*2)defercancel()_= chromedp.SetAttributeValue(formNodes,"target", nameStr, chromedp.ByNodeID).Do(tCtx)}
要成功的提交表单,就得正确触发表单的submit操作。不是所有的前端内容都有规范的表单格式,或许有一些form连个button都没有,所以这里有三种思路可供尝试,保险起见建议全部都运行一次:
•在form节点的子节点内寻找type=submit的节点,执行elementHandle.click()方法。•直接对form节点执行JS语句:form.submit(),注意,如果form内有包含属性值name=submit的节点,将会抛出异常,所以注意捕获异常。•在form节点的子节点内寻找所有button节点,全部执行一次elementHandle.click()方法。因为我们之前已经重定义并锁定了表单重置函数,所以不用担心会清空表单。
这样,绝大部分表单我们都能触发了。
/点击按钮type=submit/func(tab Tab) clickSubmit() {defertab.formSubmitWG.Done()// 首先点击按钮type=submitctx:= tab.GetExecutor()// 获取所有的form节点 直接执行submitformNodes, formErr := tab.GetNodeIDs(form)ifformErr !=nil|| len(formNodes) ==0{logger.Logger.Debug("clickSubmit:getformelementerr")ifformErr !=nil{logger.Logger.Debug(formErr)}return}tCtx1, cancel1 := context.WithTimeout(ctx,time.Second2)defercancel1()_= chromedp.Submit(formNodes, chromedp.ByNodeID).Do(tCtx1)// 获取所有的input标签inputNodes, inputErr := tab.GetNodeIDs(form input[type=submit])ifinputErr !=nil|| len(inputNodes) ==0{logger.Logger.Debug("clickSubmit:getforminputelementerr")ifinputErr !=nil{logger.Logger.Debug(inputErr)}return}tCtx2, cancel2 := context.WithTimeout(ctx,time.Second2)defercancel2()_= chromedp.Click(inputNodes, chromedp.ByNodeID).Do(tCtx2)}/clickallbutton/func(tab Tab) clickAllButton() {defertab.formSubmitWG.Done()// 获取所有的form中的button节点ctx:= tab.GetExecutor()// 获取所有的button标签btnNodeIDs, bErr := tab.GetNodeIDs(form button)ifbErr !=nil|| len(btnNodeIDs) ==0{logger.Logger.Debug("clickAllButton:getformbuttonelementerr")ifbErr !=nil{logger.Logger.Debug(bErr)}return}tCtx, cancel1 := context.WithTimeout(ctx,time.Second2)defercancel1()_= chromedp.Click(btnNodeIDs, chromedp.ByNodeID).Do(tCtx)// 使用JS的click方法进行点击varbtnNodes[]cdp.NodetCtx2, cancel2 := context.WithTimeout(ctx,time.Second*2)defercancel2()err:= chromedp.Nodes(btnNodeIDs, &btnNodes, chromedp.ByNodeID).Do(tCtx2)iferr !=nil{return}for_, node := range btnNodes {_= tab.EvaluateWithNode(js.FormNodeClickJS, node)}}
事件触发
1.对JavaScript协议的内联事件触发,执行以下js
(asyncfunctionclick_all_a_tag_javascript(){letnodeListHref =document.querySelectorAll("[href]");nodeListHref =window.randArr(nodeListHref);for(letnodeofnodeListHref) {letattrValue = node.getAttribute("href");if(attrValue.toLocaleLowerCase().startsWith("javascript:")){awaitwindow.sleep(%f);try{eval(attrValue.substring(11));}catch{}}}letnodeListSrc =document.querySelectorAll("[src]");nodeListSrc =window.randArr(nodeListSrc);for(letnodeofnodeListSrc) {letattrValue = node.getAttribute("src");if(attrValue.toLocaleLowerCase().startsWith("javascript:")){awaitwindow.sleep(%f);try{eval(attrValue.substring(11));}catch{}}}})()
2.对常见的内联事件触发
(asyncfunctiontrigger_all_inline_event(){leteventNames = ["onabort","onblur","onchange","onclick","ondblclick","onerror","onfocus","onkeydown","onkeypress","onkeyup","onload","onmousedown","onmousemove","onmouseout","onmouseover","onmouseup","onreset","onresize","onselect","onsubmit","onunload"];for(leteventNameofeventNames) {letevent= eventName.replace("on","");letnodeList =document.querySelectorAll("["+ eventName +"]");if(nodeList.length>100) {nodeList= nodeList.slice(0,100);}nodeList =window.randArr(nodeList);for(letnodeofnodeList) {awaitwindow.sleep(%f);letevt =document.createEvent('CustomEvent');evt.initCustomEvent(event,false,true,null);try{node.dispatchEvent(evt);}catch{}}}})()
3.对之前hook的事件触发,对于某些节点,可能会存在子节点也响应的事件,为了性能考虑,可以将层数控制到三层,且对兄弟节点随机选择一个触发。简单画图说明:
1.
(asyncfunctiontrigger_all_dom2_custom_event(){functiontransmit_child(node, event, loop){let_loop = loop + 1if(_loop > 4) {return;}if(node.nodeType === 1) {if(node.hasChildNodes) {letindex =parseInt(Math.random()*node.children.length,10);try{node.children[index].dispatchEvent(event);}catch(e) {}let max = node.children.length>5?5:node.children.length;for(letcount=0;count200) {nodes= nodes.slice(0,200);}nodes =window.randArr(nodes);for(letnodeofnodes) {letloop = 0;awaitwindow.sleep(%f);letevent_name_list = node.getAttribute("sec_auto_dom2_event_flag").split("|");letevent_name_set =newSet(event_name_list);event_name_list = [...event_name_set];for(letevent_nameofevent_name_list) {letevt =document.createEvent('CustomEvent');evt.initCustomEvent(event_name,true,true,null);if(event_name =="click"|| event_name =="focus"|| event_name =="mouseover"|| event_name =="select") {transmit_child(node, evt,loop);}if( (node.className && node.className.includes("close")) || (node.id && node.id.includes("close"))) {continue;}try{node.dispatchEvent(evt);}catch(e) {}}}})()
4.监控插入的节点,如果新增节点的href src含有JavaScript协议,则手动触发。这似乎会漏一些内联事件的触发。(functioninit_observer_sec_auto_b(){window.dom_listener_func_sec_auto =function(e){letnode = e.target;letnodeListSrc = node.querySelectorAll("[src]");for(leteachofnodeListSrc) {if(each.src) {window.addLink(each.src,"DOM");letattrValue = each.getAttribute("src");if(attrValue.toLocaleLowerCase().startsWith("javascript:")){try{eval(attrValue.substring(11));}catch{}}}}letnodeListHref = node.querySelectorAll("[href]");nodeListHref =window.randArr(nodeListHref);for(leteachofnodeListHref) {if(each.href) {window.addLink(each.href,"DOM");letattrValue = each.getAttribute("href");if(attrValue.toLocaleLowerCase().startsWith("javascript:")){try{eval(attrValue.substring(11));}catch{}}}}};document.addEventListener('DOMNodeInserted',window.dom_listener_func_sec_auto,true);document.addEventListener('DOMSubtreeModified',window.dom_listener_func_sec_auto,true);document.addEventListener('DOMNodeInsertedIntoDocument',window.dom_listener_func_sec_auto,true);document.addEventListener('DOMAttrModified',window.dom_listener_func_sec_auto,true);})()
窗口阻塞处理
crawler处理了 alert()/prompt() 基础认证等等的阻塞。
chromedp.ListenTarget(tab.Ctx,func(vinterface{}) {switch v := v.(type) {//casenetwork.EventLoadingFailed:// logger.Logger.Error("EventLoadingFailed ", v.ErrorText)// 401 407 要求认证 此时会阻塞当前页面 需要处理解决casefetch.EventAuthRequired:tab.WG.Add(1)gotab.HandleAuthRequired(v)// close Dialogcasepage.EventJavascriptDialogOpening:tab.WG.Add(1)gotab.dismissDialog()}})
但是还有 打印 和 文件上传窗口可能阻塞窗口
打印事件可以hook函数,文件上传窗口可以用Page.setInterceptFileChooserDialog过滤。
End
一些还可以优化的部分,表单填充可以识别参数长度max-length、min-length
从Crawlergo的设计和源码中能提取出很多东西来,
•基于网页结构的大量网页快速相似匹配,如果能集成到那些网络空间引擎中应该会很好玩,但似乎还没有一家做过。
有了原生的动态爬虫支持,对自动化漏扫也有了更多的想法,例如通过hook一些触发函数,污点检测来检测dom xss,爬虫的原始请求包可以直接推到w13scan中。有了自动化爬虫,后续所有流量都可以存储一份,直接用搜索语法来找到相同参数的页面进行poc测试等等。。
作者的代码风格太不go了,想重写一份了。
参考
•https://www.anquanke.com/post/id/178339#h2-17•https://xz.aliyun.com/t/7064#toc-11
广告
后续会在知识星球开源相关部分代码,欢迎加入我的知识星球。公众号回复“知识星球” 即可获得星球详情以及加入链接。