#indiewebcamp 2014-02-15

2014-02-15 UTC
#
Jeena
hm or is it some extencion of mine? I have investigated that once already but I don't think I was able to find a solution.
#
snarfed
might be your setup
#
snarfed
the https facebook and twitter avatars here render for me in firefox: https://snarfed.org/2014-01-16_coffee-the-third-most-popular-stimulant-in-the-neighborhood
#
Jeena
just restarted my firefox without addons and they show up here too now
#
Jeena
ah it was the Disconnect addon, which kind of does exactly what it was installed to do, hehe damn
tilgovi, hallettj, CheckDavid, paulcp, pasevin, gRegor`, nloadholtes, snarfed, bnvk, jedahan and skinny joined the channel
#
KartikPrabhu
tommorris: was lookinf through the mf2py code in hopes of contributing to it. Any reason you use html5lib instead of beautiful soup? The latter already has some features that you have sough to implement like parsing multiple attribute values like classnames
bnvk, snarfed, squeakytoy, jedahan and squeakytoy2 joined the channel
#
tommorris
KartikPrabhu: I’m not sure BeautifulSoup has good HTML5 support.
#
tommorris
but if BeautifulSoup does the job, I’m happy to switch
#
KartikPrabhu
interesting! I just finished adding a lot of comments while reading the parsing code. Is it possible that you (or I) put this on github? (I don't quite have the hang of mercurial yet! :D )
eschnou, tantek, KartikPrabhu, indiewebcamp-vis, bnvk, ttepasse, glennjones, jedahan, Kopfstein, pasevin, melvster, brianloveswords and snarfed joined the channel
#
snarfed
morning indiewebcamp
snarfed joined the channel
#
snarfed
twitter's streaming api is getting to be more trouble than it's worth, at least for bridgy
#
snarfed
it stared 401ing all of my connections a week ago. i've done a lot of digging, but haven't figured out why yet
#
snarfed
i only need it for favorites, and i'm about to switch to scraping to get those :/
#
bear
your using the control_uri / user_uri streaming method?
#
snarfed
user streams
#
bear
are you cycling thru multiple IPs per oauth id?
#
bear
they have maximums for IPs *and* connections
#
snarfed
yeah, i've been looking at that. i believe i cycle through IPs, but only a few per day at most
#
snarfed
and 53 users right now, which as far as i can tell is comfortably under their limit
#
bear
the problem is that twitter has never said how often they flush the IP bucket
#
bear
and 50 is one of those numbers i have never found in the docs but seems to be a magic user stream number
#
bear
they really want folks to use site streams
#
bear
ah - the appengine twitter lib (I was wondering which twitter lib you used)
#
snarfed
i use tweepy
#
snarfed
any chance you remember where you saw the 50 number?
#
bear
in a post to twitter dev like a year ago
#
snarfed
right now it 401s for me when deployed even when i only try to connect one user, but that may be left over from triggering it with >50
#
bear
ah tweepy - cool
#
bear
<-- author (one of) of twitter-python
#
snarfed
ah, cool!
#
snarfed
glad you saw my messages :P
#
snarfed
scraping is probably the answer, then
#
snarfed
i'd love to sign up for site streams, but i worry about attracting any human attention, since i suspect they'd shut it down
#
bear
i'm watching the bridgy repo :)
#
snarfed
thanks!
#
bear
yea, it's a fine line that many have crossed once you sign up for site streams
#
snarfed
yup. happened to me with another app a bit ago, even without site streams. https://snarfed.org/2013-07-09_twitter-atom-is-back
#
bear
the worse problem for user streams i've found is that it's sampled output
#
snarfed
happened around 1k users, which may be a threshold for human reviewing
#
snarfed
really? i think i was seeing complete data with user streams on bridgy
#
bear
it's fine as long as the users your tracking *and* the total output is low
#
snarfed
got it
#
bear
but once you get someone really popular then they start dropping data
#
snarfed
do you think site streams is better?
#
bear
it's much better - but the hoops you have to jump thru are tough
#
bear
that's why so many stick to user streams and just have multiple workers
#
bear
each with different oauth keys and IPs
#
snarfed
yeah. i worked around getting banned by asking users to geneate their own app key
#
snarfed
awful UX, but oh well
#
bear
scraping is sometimes faster because they put so much data in the page response
#
snarfed
ah, rather than a bunch of api calls
#
snarfed
it's so frustrating that you can't get favorites from the rest api
#
snarfed
at least not favs of a tweet
#
snarfed
the rest of the api is so complete, i don't get why that's not there
#
bear
a lot of the rest api stopped growing once they got streams and were taken over by business people
#
bear
(IMO)
#
bear
i've taken to archiving my twitter emails - because I get one everytime someone favorites my posts
#
bear
to backfill what I miss from the user stream
#
snarfed
oh wow, interesting
#
snarfed
what's the app? or use case?
#
bear
just my personal testing tool
#
bear
always making sure python-twitter is working
bnvk joined the channel
#
bear
yea, the sad part is that the twitter web client does exactly what I want to do - return a list of people who fav'd a post
#
snarfed
right!
#
snarfed
or /i/activity/favorited_popup , which returns json
#
bear
nods
#
snarfed
either way
snarfed joined the channel
#
bear
snarfed - question, are you pulling out the event notifications from the stream as they happen?
#
snarfed
bear: yes
#
bear
ah - ok
#
bear
yea, then you are doing everything I would do - just twitter isn't giving you the data flow it should be
#
snarfed
you mean, when it 401s?
#
snarfed
thanks for looking
#
bear
snarfed - yes, they shutdown all v1 API first week of february
barnabywalters joined the channel
#
snarfed
ah, good to know
barnabywalters_, barnabywalters, saeidw, tilgovi, snarfed, eschnou and caseorganic joined the channel
#
aaronpk
!tell barnabywalters there's something strange with the markup here. Check it out on the pin13 parser http://pin13.net/mf2/?url=http%3A%2F%2Fwaterpigs.co.uk%2Farticles%2Fhalsway-gurdy-weekend-2014%2F
#
Loqi
Ok, I'll tell him that when I see him next
Acidnerd and eschnou joined the channel
#
caseorganic.com
edited /site-deaths (+19) "/* Site Deaths */ Added link to site changes page under See Also"
(view diff)
#
caseorganic.com
created /site-changes (+1923) "Created page. Added sections, structure and content. Used the template of site-deaths"
(view diff)
snarfed joined the channel
#
caseorganic.com
edited /site-changes (+2359) "/* Site Changes */ Added stubs for new examples."
(view diff)
#
caseorganic
Yes! Site Changes! Have been wanting to make that page for a long time on cyborganthropology.com. Makes much more sense on IndieWebCamp wiki
#
caseorganic.com
edited /why (+40) "/* Identity loss */ Added site changes"
(view diff)
#
caseorganic.com
edited /stable-pages (+19) "/* Stable pages */ Added site changes"
(view diff)
#
snarfed
hey bear, do you know of any better way to scrape twitter favorites than /i/activity/favorited_popup? it cuts off at 25. :/
#
snarfed
(/i/tweet/html doesn't have favs at all)
#
caseorganic.com
edited /stable-pages (+15) "/* Stable pages */ added site changes and timeline"
(view diff)
#
caseorganic.com
edited /silo (+19) "/* See also */ Added site changes"
(view diff)
#
caseorganic.com
edited /NeoCities (-2) "fixed grammer"
(view diff)
#
aaronpk
snarfed: you enabled twitter cards on your site!
#
snarfed
aaronpk: yup! been there for a while
#
aaronpk
just saw cause one of tantek's tweets embedded your site's card
#
snarfed
i don't love them, since they somehow get the raw markdown, before it's rendered…but probably still a net win
#
snarfed
yours look good!
#
aaronpk
yeah it's definitely a hack. I'd love to embed other page's summaries by parsing the microformats on ppls' sites!
#
caseorganic.com
edited /history (+19) "/* See Also */ Added site changes"
(view diff)
#
aaronpk
thanks! mine are ok except the default image is my face which is sometimes a bit much
#
snarfed
that's my single biggest motivation for not liking invisible metadata
#
snarfed
i want my picture in my home page's h-card, but i *don't* want my picture there
#
snarfed
er, sorry, motivation for *liking* it. or at least tolerating it
caseorganic joined the channel
#
@TassianaBeta
RT @t: Two wonderful talks on #privacy #ownyourdata #indieweb by @caseorganic: http://www.youtube.com/ http://vimeo.com/86329113 (ttk.me t4UZ1)
(twitter.com/_/status/434800047025553408)
#
@TassianaBeta
RT @t: Sign-up page is up for the next week's Homebrew Website Club meetup @MozSF & @ESRIPDX: http://indiewebcamp.com/events/2014-02-12-homebrew-website-club (ttk.me t4US1)
(twitter.com/_/status/434800062825517057)
#
@TassianaBeta
RT @t: @benwerd has offered handmade art with a twist: write an #indieweb reply (like this) & send a webmention. I want one! (ttk.me t4UL1)
(twitter.com/_/status/434800076297629696)
#
@TassianaBeta
RT @t: much #IndieWeb Bridgy & #IndieAuth progress! Come to 6:30pm Homebrew Website Club in SF & Portland TONIGHT: http://tantek.com/2014/029/t1/indieweb-progress-bridgy-indieauth-homebrew-tonight
(twitter.com/_/status/434800079036481536)
#
@TassianaBeta
RT @t: going to @benwerd's Introduction to the IndieWeb talk late http://werd.io/2014/introduction-to-the-indieweb had another event before. (ttk.me t4UB2)
(twitter.com/_/status/434800087274123264)
#
@TassianaBeta
RT @t: @adactio @jkphl no inferred p-author. Even better, there's the page authorship building block: http://indiewebcamp.com/authorship (ttk.me t4U51)
(twitter.com/_/status/434800114818105345)
#
@TassianaBeta
RT @t: First Homebrew Website Club meetup this year is in one week 2014-015 18:30 both @MozSF @EsriPDX http://indiewebcamp.com/events/2014-01-15-homebrew-website-club (ttk.me t4Ty1)
(twitter.com/_/status/434800132971053056)
#
Loqi
[@TassianaBeta] RT @t: 2013 #indieweb pieces: IndieAuth comments webmention events Bridgy
#
Loqi
2014 together invincible. ♫ http://vimeo.com/40111334 (ttk.me t4Tu1) (http://twtr.io/i8gGB78QPU)
#
aaronpk
is suspicious of people who retweets all of tantek's recent tweets
#
caseorganic
aaronpk: yes
#
aaronpk
!spammer TassianaBeta
#
Loqi
Got it! There are now 8 spammers blacklisted
lmjabreu joined the channel
#
aaronpk
oh I turned off mention notifications here after that bridgy incident
#
aaronpk
forgot about that
#
caseorganic.com
created /events/2014-02-26-homebrew-website-club (+2910) "Created meeting page for the next Homebrew Website Club meetup!"
(view diff)
#
snarfed
aaronpk: apologies again for that flood
#
aaronpk
was fun
#
caseorganic.com
edited /Main_Page (+1) "/* Homebrew Website Club */ Changed page to show next meeting date"
(view diff)
#
snarfed
i think we ended up deciding to keep invitee mentions though
#
snarfed
so it could easily happen again with the next big fb event
#
snarfed
i'm guessing it would take a bit of work to suppress only invite rsvp webmentions?
#
snarfed
(in loqi)
#
aaronpk
yeah i'm not doing too much parsing of the actual page content, but that's a reasonable thing to add
#
snarfed
only send the first mention of a page? you mean, of a source?
#
snarfed
good idea, but wouldn't prevent the flood specifically. each invite has its own bridgy url :/
snarfed joined the channel
skinny joined the channel
#
aaronpk
yes, but it would prevent the little flood above :)
#
snarfed
ah. sure!
caseorganic and ttepasse joined the channel
snarfed joined the channel
#
aaronparecki.com
created /principles-de (+4535) "Copy to German page for translation"
(view diff)
caseorganic joined the channel
#
aaronparecki.com
edited /principles-de () "(-700) Edits to remove "offensive language""
(view diff)