 Andrew Cantino
|
fd2e8cd8b6 |
add headers option to WebsiteAgent |
10 anos atrás |
 Akinori MUSHA
|
e8751af629 |
Add a `user_agent` option to WebsiteAgent. |
10 anos atrás |
|
Akinori MUSHA
|
85a7369e65 |
Use Faraday in WebsiteAgent and make HTTP backend library selectable. |
10 anos atrás |
 Maximilian Clarke
|
9bf3c2c824 |
Updated WebsiteAgent to receive events |
10 anos atrás |
|
Maximilian Clarke
|
19c005fe45 |
Modified website_agent to take an array of urls |
10 anos atrás |
 Andrew Cantino
|
7d9279b871 |
Merge pull request #212 from knu/website_agent-force_encoding |
10 anos atrás |
|
Akinori MUSHA
|
8ea2ba573f |
Add :xpath support to WebsiteAgent. |
10 anos atrás |
|
Akinori MUSHA
|
7bc20a0b44 |
Add :force_encoding support to WebsiteAgent. |
10 anos atrás |
|
Andrew Cantino
|
99644a426d |
Add XKCD hovertext to default seed and website agent |
10 anos atrás |
|
Andrew Cantino
|
f4bae10250 |
minor code cleanup |
10 anos atrás |
 Alex Piggott
|
b1898cc7ff |
#154 Improvements to website deduplication logic |
10 anos atrás |
|
Alex Piggott
|
7b38df61ed |
#135 #141 2 deduplication fixes for the website agent |
10 anos atrás |
 Albert Sun
|
7996954a3b |
add a `basic_auth` option to the website agent |
10 anos atrás |
|
Andrew Cantino
|
9c48338347 |
fix specs |
11 anos atrás |
|
Andrew Cantino
|
00b7423dd7 |
add cached columns for event creation and last errors, reducing the number of SQL queries |
11 anos atrás |
|
Albert Sun
|
43194c3c1b |
in website agent with type json, allow extract to be blank; in which case, the entire json object will be stored as the payload |
11 anos atrás |
|
Andrew Cantino
|
7372244d0f |
return false from working? when an agent's most recent log is an error |
11 anos atrás |
|
Andrew Cantino
|
00727fbd4d |
add Agent Logs; add logging to WebsiteAgent; refactor flash notices and add event notices |
11 anos atrás |
 itkevin
|
fd8761177f |
When crawling websites tith relative URLs, make them absolute |
11 anos atrás |
 Kevin Lindecke
|
cd7f23aa50 |
Added test for relative paths |
11 anos atrás |
|
Andrew Cantino
|
b876759b7b |
Add JSONPath for hash paths and add JSON parsing to the WebsiteAgent. |
11 anos atrás |
|
Andrew Cantino
|
620acffa5a |
initial commit |
11 anos atrás |
Andrew Cantino
Akinori MUSHA
Maximilian Clarke
Andrew Cantino
Alex Piggott
Albert Sun
itkevin
Kevin Lindecke