 Akinori MUSHA
|
f8a6230447 |
Switch faraday_middleware to the upstream master |
9 years ago |
|
Akinori MUSHA
|
7bcb5e1374 |
Exclude `deflate` from Accept-Encoding by default |
9 years ago |
 Andrew Cantino
|
48d5990146 |
Allow url_from_event in the WebsiteAgent to be an Array |
9 years ago |
|
Akinori MUSHA
|
2caf45ab7d |
Use FaradayMiddleware::Gzip in faraday_middleware 0.10.0 |
9 years ago |
 Andrew Cantino
|
8d1a05fb7c |
fix specs |
9 years ago |
|
Andrew Cantino
|
4f319a3ab4 |
Merge branch 'master' of https://github.com/stvnrlly/huginn into stvnrlly-master |
9 years ago |
|
Andrew Cantino
|
3584c8fa5e |
allow integer indexes as strings |
9 years ago |
|
Akinori MUSHA
|
1f49085770 |
Fix a spec for an error message |
9 years ago |
|
Akinori MUSHA
|
130ca6c9af |
Rename the option name to `url_from_event`, suggested by @cantino |
9 years ago |
|
Akinori MUSHA
|
1e336f029e |
Add a spec for the url_on_receive option |
9 years ago |
|
Akinori MUSHA
|
e791f75648 |
Add a new option `use_namespaces` to WebsiteAgent. |
10 years ago |
|
Akinori MUSHA
|
2bb89c4e39 |
Add specs for the extraction type `xml`. |
10 years ago |
|
Andrew Cantino
|
84ebfa18cb |
add a validation that warns the user if they have not provided a path when using JSONPath |
9 years ago |
 stvnrlly
|
5e1730351f |
update some spec scripts to new cleanup time |
9 years ago |
|
stvnrlly
|
795050636f |
add spec for website unzip |
9 years ago |
 Dominik Sander
|
0daea40bdf |
On postgres queries without an ORDER are ordered randomly, this caused |
9 years ago |
 Chris Eidhof
|
b2218df623 |
Added merge mode |
10 years ago |
|
Andrew Cantino
|
991e1466c6 |
fix spec when user agent is different |
10 years ago |
|
Akinori MUSHA
|
5031cbbbac |
Migrate to RSpec's new expect syntax using Transpec. |
10 years ago |
|
Akinori MUSHA
|
c21bada226 |
WebsiteAgent: Provide a variable _response_ for interpolation. |
10 years ago |
|
Akinori MUSHA
|
df907c0290 |
Extend the spec to give a better idea on how to use `to_xpath`. |
10 years ago |
|
Akinori MUSHA
|
0c490aa82d |
Add a Liquid filter `to_xpath`, which quotes a string for use in XPath expression. |
10 years ago |
|
Akinori MUSHA
|
863e2b8c70 |
WebsiteAgent should interpolate values from incoming event payload. |
10 years ago |
|
Akinori MUSHA
|
fca8051e81 |
Add a parser type `text` to WebsiteAgent. |
10 years ago |
|
Akinori MUSHA
|
7b6119f1f2 |
`"text": true` should have meant ".//text()", not "text()". |
10 years ago |
|
Akinori MUSHA
|
a800342c29 |
WebsiteAgent: Add a spec for XPath returning an integer value. |
10 years ago |
|
Akinori MUSHA
|
4d623c5893 |
WebsiteAgent: Introduce per-node XPath evaluation in extraction. |
10 years ago |
|
Andrew Cantino
|
f4df522f2f |
adding a basic RSS agent |
10 years ago |
|
Andrew Cantino
|
fd2e8cd8b6 |
add headers option to WebsiteAgent |
10 years ago |
|
Akinori MUSHA
|
e8751af629 |
Add a `user_agent` option to WebsiteAgent. |
10 years ago |
|
Akinori MUSHA
|
85a7369e65 |
Use Faraday in WebsiteAgent and make HTTP backend library selectable. |
10 years ago |
 Maximilian Clarke
|
9bf3c2c824 |
Updated WebsiteAgent to receive events |
10 years ago |
|
Maximilian Clarke
|
19c005fe45 |
Modified website_agent to take an array of urls |
10 years ago |
|
Andrew Cantino
|
7d9279b871 |
Merge pull request #212 from knu/website_agent-force_encoding |
10 years ago |
|
Akinori MUSHA
|
8ea2ba573f |
Add :xpath support to WebsiteAgent. |
10 years ago |
|
Akinori MUSHA
|
7bc20a0b44 |
Add :force_encoding support to WebsiteAgent. |
10 years ago |
|
Andrew Cantino
|
99644a426d |
Add XKCD hovertext to default seed and website agent |
10 years ago |
|
Andrew Cantino
|
f4bae10250 |
minor code cleanup |
10 years ago |
 Alex Piggott
|
b1898cc7ff |
#154 Improvements to website deduplication logic |
10 years ago |
|
Alex Piggott
|
7b38df61ed |
#135 #141 2 deduplication fixes for the website agent |
10 years ago |
 Albert Sun
|
7996954a3b |
add a `basic_auth` option to the website agent |
10 years ago |
|
Andrew Cantino
|
9c48338347 |
fix specs |
11 years ago |
|
Andrew Cantino
|
00b7423dd7 |
add cached columns for event creation and last errors, reducing the number of SQL queries |
11 years ago |
|
Albert Sun
|
43194c3c1b |
in website agent with type json, allow extract to be blank; in which case, the entire json object will be stored as the payload |
11 years ago |
|
Andrew Cantino
|
7372244d0f |
return false from working? when an agent's most recent log is an error |
11 years ago |
|
Andrew Cantino
|
00727fbd4d |
add Agent Logs; add logging to WebsiteAgent; refactor flash notices and add event notices |
11 years ago |
 itkevin
|
fd8761177f |
When crawling websites tith relative URLs, make them absolute |
11 years ago |
 Kevin Lindecke
|
cd7f23aa50 |
Added test for relative paths |
11 years ago |
|
Andrew Cantino
|
b876759b7b |
Add JSONPath for hash paths and add JSON parsing to the WebsiteAgent. |
11 years ago |
|
Andrew Cantino
|
620acffa5a |
initial commit |
11 years ago |
Akinori MUSHA
Andrew Cantino
Andrew Cantino
stvnrlly
Dominik Sander
Chris Eidhof
Maximilian Clarke
Alex Piggott
Albert Sun
itkevin
Kevin Lindecke