Check out the video updates at LBRY.TV
*'s direly important for site opperation
More to be added as I find them.
What is still broken as of 02/18/2021: top links, add feed, search.
suggestions, thoughts or coffee can be submitted to joe [at] shellcity [dot] net as long as you inject them with bleech first. Replace [at] and [dot] with @ and .
considering open sourcing this at a later time. Not to sure yet.
- Domain Changes
*Domain Transfer Two steps, one done
- Hosting Transfer
Move Virtual Machine from old host to new
Restore from previous backup (and have it fail because of incompatible code)
upgrade virtual machine when out of testing
- Code update
Identify code that needs updated
Update provider database to remove dead feeds. UPDATE: timed update task is getting a list
Update feeds database
*Rewrite DB connector/queries (~40% done).
*redo feed updating to a timed task, from on demand for faster site loading update so many every 5 minutes based on last update time UPDATE: inital testing seems to be working
Change feed update from third party to internal functions
Verify user tables still work with updated connectors
- Top XX items: Remove? Keep? update daily? UPDATE: rewriting for.. bigger reasons than wanting it or not
- Search/All the Latest, needed? likely just remove at least short term same as top XXX list
- New problems 12/28/20
- Users, db feed splitting into new feed system. Not looking forward to this
formatting feed items for intergration in to current site
- New 12/30/20
- NEW 01/04/21
Database speed aweful. Could be the lousy VM 1c 1gb large with large tables. Optimization/Caching maybe required
- NEW 01/24/21
- Been having thoughts on privacy. Decided after finished if i can find another analyics service i will set all logs to wipe weekly. Not that there is an issue just, why not?
Doing first major OS patch with the "working" site. "Hold my beer" so the saying goes....
Finally updater working.
I got some personal projects i'm working on along with my 9-5 job so in the near term updates will be limited.
this week i hope to be closer to the top XX working. and updating some links previously going to Bob.
At the end of all of this, besides the actual UI... about 90% of the site will have been re-written into much leaner and updated code.
In a braindead sleep depraved moment i got an idea and it worked.. but i disable the updater and just realized tonight i never started it again.... .. its starting ... wait for it...
Feeds and updating are working. found three that have never updated. Boradband reports, space daily and DV hardware. there are a couple that i don't think are updating either
Next project getting the add feed to work. should be easy. the function exists in a include that no longer exists.
after that top xxx. I believe these where done with flat files before but think i'm going to create a new table for them for easyier searching
I honestly don't know what this did before but shouldn't be to hard once everything gets added to the database. this and add feed may not work with the current page format.
It may not happen this week. I'm working on another personal project. Never learned to do it so i'm learning revers_proxy. I never used docker because of port numbers that are blocked. this takes care of that for me.
Along with that, I setup room in Matrix using Element. for those that don't know Element is a more secure and Private messaging platfom similar to discord just private. no email address required.
New video should be going up today going over this weeks progress.
For those wondering. The update code was reduced to about 200 lines including comments.
added a bit of error checking last night and logging. Need to clean that up some to get easier to read info.
quite a few feeds are still broke but not the fault of DR. either RSS for the site no longer exists or moved. Checking ones that pop out at me and updating urls where possible.
last night the update completly failed but had to sleep. error is i was returning in the wrong spot. quick 1 minute fix
re-running an update to all feeds now.
Decriptions now work on the feeds for almost all. some are failing on insert. Need to put a check on that to blank out the description if error.
once that is done i'm going to work on the top feeds. after a quick look I think i'm going to rewrite it for more modern code. It might be as simple as a permissions err.
Okay all set with what works which seems to be 90% or more of the feeds. going to pick through the remainder some other time.
yup, i'm rewriting the top feeds. I just don't know perl enough to make semi-major changes. I think it threw alot of errors before like it is now so maybe it is working and just won't really update until enough traffic comes through.
Either way it needs rewritten. I found some stuff in here from before that isn't acceptable in 2021.. 2010 sure.
But no matter what it will perform better :-)
running one last update before i put it to auto updateing. Note it might be a little wierd for a bit. Not all feeds update per fucntion but every feed "should" update every 3hrs or so.
I'm going to work on other projects for a few days, but if there is something specific or feed you would like to see let me know and i can come back to it.
set to run at lunch, WORKING GREAT!
Still need to fix error checking massivly but so far so good.
Many more feeds that previously didn't load are not loading.
ALL OF THE DEFAULT PAGE now works.
ALL but a handfull of the recommended works. A few i know why and may or may not be removed. TBD
THe updater IS slower but i am also letting it parse more so.. +/- Still updates 1k in under 30minutes. so..
I'll work on it some more tonight
WHOOPS! I aranged all the feeds back-ass-wards hahahah should be fixed the next time I run an update
updating this late as i'm waiting on stuff to finish at the office
Updated a bunch of feeds and started flaggin feeds that don't seem to exist any more.
This has got me wondering if some of the old feeds i removed might still be valid. Will revisit this later (i still have them as i thought this might possibly happen)
I'm on to v4.0.3 of the updater. again found what i think is a better way.
As they say, "Hold my beer"
Guess what I did yesterday and a bit today? wll did a vid on the lbry.tv page and rewrote the entire update class
why? speed mostly. the previous improvments got me some ideas.. and also made error checking easier.
the class is much smaller but the calling script is a bit bigger. I suppose i could put that in the class too but that would get overly complicated.
hahah! that didn't work think i know why....
leaveing old updater for now.
Any thoughts about having a community chat group? Let me know what you think!
All Negative, I'm free tomorrow!
Netcraft had two near identical entries except for the description and id. one worked and one didn't.
"MOST" fo the recommended feeds (select able at the bottom) work
only a crap ton more to fix. i have a feeling they're similar problems.
Tests hopefully return tonight (cross fingers)
found a way to speed up the updating which inturn sped up the front page. Got a few other ideas to speed up/reduce the amount of queries. Over 6k queries an hour!
changed the update flow again every 30 minutes it updates a portion. I need to tinker with this a bit to make sure every feed gets hit but it will get there.
setup an internal stats setup. If it works half way decently going to remove Google. Initial look shows way more people looking at this site daily than i realized. Sorry i'll try to get stuff done faster! as always if you find something you specifically want working mail me and i'll put it on my list!
Hopefully will get the error checking done today to find out why those feeds are not updating.
cleaned up the queries MUCH better now.
I still keep bouncing around the some feeds not updating thing. ran into something that might be doing it... but.. i'm going to bounce around that again for the night ha!
One tested Negative but need to do more tests! man what a pain.
Other than the ongoing issue with some of the feeds they seem to be working, updating every 4 hours now
Not been doing Much. Family still on the COVID watch and this hasn't been a priority though still keeping thoughts on it.
Thanks to Ed K. for letting me know about the 404 error. forcing ssl messed up my rewrite.
Speaking of, Addled ssl requirement. Google Analitics, add this, etc.. cause it to be n0t 100% becuase of external files.
Will probably remove these, privacy issues. ::middle_finger:: google will be looking for another non-evasive statistic service
Some feeds with valid feeds are still not updating. going to create better error checking and logging for these and go from there.
Next priority is the top lists. I have no idea how this really works... yet
While I havn't made much site progress i've made some other external progress. More to come.
Waiting on another COVID test to come through. Third time i've been isolated >.<
During this time i found what i think is emptying the database. going to start up the cronjob for updating again.
installing updates for the os, hopeing it doesn't break shit. Thankfully I have backups.
Over the weekend all the feeds broke again. I cannot find a reason for it. there is nothing in the old site set to update or delete the table the feed items are in
only two things i can think of is something in my logic is deleting them. I disabled the delete function. seems to have worked for a day.
other thing i can think of, I was borderline with vm resources and it is just breaking. I upgraded the VM.
Need to get the number of queries down and also disc reads. However i believe this will also break the old code.
Scarlet Vision comes out on Diz+ today! got to binge mando to get to it.
In pushing to update the default feeds today.
Trying to avoid politics on recommended or default. It amazes me how Politices has infeltrated Tech.
It gets even worse when you realize its is so one sided. The truth doesn't matter any more... /sigh.. anyway thats my statement on that.. expect very little of it Geek before spreading the word of liberty here ;-)
Once i get the default done I think all i really need to do is fix search and top lists.
For the feeds.. since sooo many of your feeds are gone I am going to put ALL feeds in the recommended section for a month or so. YOu may see some politices there that you don't agree with... suck it up butter cup. Half of america doesn't agree with you and you know what? That is okay different opinions does not make someone bad.... oh shit.. thats another statement.. i lied sorry.
Thanks to everyone stick it out. I'm getting there.
Default done, need to remove one or two it seems. monitoring
Just realized how long its been since i updated shell city. I really need to remember.
found some feeds not loading did some error checking.. broke them all. working again but still those handful not.
other wise seems to be okay. might mess around with that more over the weekend.
Did add some more err checking but its not picking up any errors... which.. i don't understand, there are errors >.< maybe the error is me
going to add some vm backups and then try to update the OS again.. maybe tonight but probably tomorrow X your fingers.
Yesterday found an error in the updater that was not inserting valid feeds. I got an idea, and then annother and then..... ..
I re wrote the updater this morning. it faster, more efficient and more forgiving... unless the feed it self is complete garbage.
for example daily cup of tech doesn't exist now and the suspension page really made it mad. the right error checking shoudl fix this... later.
ran an update to all remaining 900ish feeds it seems almost all worked properly.
Updated the recommended list. there are still a bunch of dupe's but getting there.
tonight or tomorrow updating the default feeds.
Got an account? let me know how your feed is looking send me screen shots or whatever If i took out one of yours let me know we'll see if we can get it back!
Busy day, nothing done that would be visible. only small changes that do not effect the main page.
Noticed some fatal error issues last night. old code popping up again. I think this might happen from time to time until i get a more complete rewrite done. (maybe never lol)
Some feeds don't display at times but works on the next. I think this is a database issue. Going to check the back end.
Ajax not working on the fly, initial testing, loging out and back in seems to correct this. might be a cookie issue.
Didn't get to remove/fix feeds last night. If nothing crazy happens to day i'll pick through them.
Investigating why some valid feeds are not updating properly in between work stuff
I think I figured out why those feeds do not appear to be updating. Mixed methods of updating/listing appear to be causing the problems on some feeds. This is a change i was planning on doing later anyway but not for this reason. I hope it doesn't break what was working lol.
After removing many that simply did not work i'm going to create a list of ones i'm removing that don't fit. this list will be at the bottom of this document.
Ever have one of those OH DUH! moments. Yeah yesterday, hit that.
Found out why my updater worked on the rewrite but not on the old page. I never disabled the old page so it was updating twice and since the old updater doesn't work on this server......
MANY feeds include media and HTML in their feeds. This is 'okay' normally but doesn't work here. Need to sanitize.
removed about another 500 feeds that where not working at all. If you found one missing that you want. you should be able to readd it useing the add link.
I removed nearly all the recommended feeds. I'll be readding a bunch. I'm guessing probably around 50 or so.
Once i get the inserts working and get recommended going I'll be working on the top list(s)
There are around 1k feeds left in the database.
Some updated policies I will generally follow and plans (my site i'll do what i want)
I will only be adding Sci/Tech too it recommended I find interesting. If you have a suggestion let me know. Sci/Tech only. YOu can add other feeds but i will not put them on the recommended.
Stuff like EFF, Internet Censorship, Net Neutrality, sec. 230 and such is good as it is at least semi related even if more political than Tech as long as they are not generally political feeds. (like huffpo tech feed okay, huffpo main feed not)
So add them, if you think you want them on recommended good ask.
I may add links to sites that maybe political in general but to articles that relate to tech and lobby for things like Net Neutrality.
DailyRotation twitter is not me. It is Bob's but feel free to talk to him, cool guy :-)
I quit most social media so I have no social media contacts. Use the contact link to get a hold of me.
Some plans on other "non mainstream" social sites but not something terribly important.
Part of the reason for not using the "big guys" is to add some variety to other networks rather than just political stuff
now that other things are working, i found some errors on inserts. Link field too small character encoding. Previous check for encoding is depricated and no longer works. May have beena problem on the old setup but i didn't notice since my involvement was just to attempt to keep the old VM up.
for some reason the link titles are coming up blank on a good portion of the links.. still.. moving forward instead of backward +1
THink i generally fixed the blank links
a handful of feeds have valid feeds yet still are not updating.
going to ride it out a while to see what happens.
Changing my auto updater task. I'll be starting it up in about an hour. It will be configured to make sure they update at least twice a day.
In the mean time i'm going to update my recommended list and update the default feeds sites that "should" be working should have this mostly done tonight.
magpi is still trying to get in there, removed all its functions (from what i can find) and removed it. Removed magpi from other modules that i could find. I have a feeling it will pop up again but one problem at a time.
If there is an RSS feed Missing that you had before let me know.
If nothing dreadful happens over night, i'm going to enable SSL. Look at optimizing the database better and consider upgrading the vm or offloading the DB to another server. DB speed isn't bad for just a few visits at a time now that i'm not updating 1k feeds at a time.
the default page is now working. HOWEVER i'm clearing the headline database and going to update all the feeds tonight.... i hope.
the TL;DR of the MAIN trouble i'm having. Old code is from 1914 and badly needs updated. New code just doesn't want to work with the layout that i don't want to change.
the good news is, the worst part with the old is the background updater.. and that is one of the things the new code is doing the best at.
I have it updating all bazzillion feeds now. Will see the front end picks them up. Site will be god aweful slow today (not that it really matters).
There is still work to be done for sure. Some of the formatting on the feeds isn't working properly. likely just stripping slashes and such.
need to clean out the feeds too. I already got about 1000 out that havn't updated in a year before the site transfer.
I "fixed" the updater to make the bad feed checker work a bit more not bad. so once it finds all the broken feeds i'll pull them out.
today i learned that the only feeds showing up in the checkboxes are the recommended ones.. there are hundreds (if not thousands) more in the data base. I mean just eyeballing it should have noticed but i didn't think about it until i started looking at the user functions.
One of the perks to doing all of this. I'm seeing a bunch of feeds for sites I have forgotten about over the years.
I'm seriously considering removing all non-tech/science stuff out as i find it. I mean i added babalon bee to the recommended.. but that was just testing. There are many that fit this bill.
I also have a plan, if the user base should return after this whole mess.... of making a list of all the feeds for users to vote on to make recommended.
Also, thoughts suggestions or just a note so i know that my time isn't being totally wasted. (i'd do it anyway though)
there are a lot of id's from recomended feeds that just don't work. I need to change my error checking script.. other wise seems much better. i also noticed that the ajax for the removal and moving of items isn't quite right. If an item is in the first slot it cannot be removed. if you move it to another slot you can remove it. might be the whole array starts at 0
My cares are gone today. scrapping what i've been working on and starting over. I missed something and i cannot find it now.
Also looks like my updater broke. 'should' be an easy fix.
figured out my problem... but my time upcoming is going to be limited. Instead of upgrading the code right now i'm going to attempt to just repopulate the database and go with the old.
Got feeds "mostly" formatted. still some minor issues, but the ajax dragging seems to be working. but don't have it configured to save it yet.
Changed Default feeds to be random 25 feeds instead of static. Gives more varity. will probably change this to be 25 random feeds for the day to reduce the query from updating every visit to query once a day.
readded a bunch of code to get the overall page format back to normal. Will have to pick through to see what I was missing later.
my current re-write of the back end got bloated to 170 lines with comments. I'm projecting about 250lines for feed updates and display and about another 1-200 for user functions.
seems like every change i make today boogers up the feed formatting.
every change is messing stuff up because I found another logic error on my part. if else for while son of a bitch lol. Once i find it things should be falling into place again as i know the other issues now.... .. i hope
Over the weekend having lots of logic errors (on my part) got no where useful
Logic starting to work for me. Got the feeds to display. Working on formatting to previous layout
formatting getting there, but seems like in the process I busted the previous CSS which might be trickling down so now i need to walk back up...
Added feed updates to cron.
working on getting feeds formatted for main page. going horribly
pretty busy day, minor things only.
running updates in the background. These updates "may" show on the main page using the old code. This will be replaced so any changes made won't be reflected.
In the processes of the updates it is watching for broken feed links and some malformed documents. also remnents from an injection attack years ago.
Guestimate of feeds to be removed, i would say around 50% which would still leave around 1000 feeds. Many of which may still be broken.
Most feeds now updated will now run around 50 random feeds every 25 minutes until live. Once live comes i'll change this to 50 every 10 minutes.
This should give the feeds better 'feel' and update more consistantly.
some progress on the userfacing side yesterday and today. will update this info later on.
12/29/2020: Latenight/early morning -> cont
feed display testing complete. Next adding a single feed to current site.
full back end functions for getting and displaying items less than 130 lines including comments! Actual calling of functions will be a bit more but not much
Domains fully transfered. Big thanks to Bob and Gerry!
DB Query update done. Will update them to "newer" implementations at a later time
Feed update done. Just need to work on the logic for the timed tasks there are MANY feeds a good portion of which no longer valid
Created a function along with the feed update to identify dead or malformed feeds.
Tables work with the new connector flawlessly. almost don't want to update the connector.
backend nearly "finished" for the feed updating
cleaning up my code to remove testing. Less than 150lines.. could be less but for another time
Feed stories cleared, starting fresh. when live not all feeds will be populated
found a more efficent way to do error checking on feeds. implemented.
feed objects breaking becuase of the error checking and malformed RSS feeds. Still a minor issue, may drop description field to remove these problems for the time being.
"assuming" the feed updating works, will work on adding it to the main page.
Error checking to find dead feeds, done. Will need to run a complete check as live comes closer
All feed requests are now using local code instead of third party.
Feeds Fixed (but may not be updating still)
- Techbargins - 1000554 - feed no longer tech bargins. Couldn't find an updated feed to the electronics/computers
- QRPSpots - 1322394 - RSS feed no longer updating from vendor. no items returned
- The Tremblin Earth - 1322091 - No updates since 2018 RSS malformed
- Wired News: Top Stories - 1000428 - Dupe
- Comments on: WIRED RSS FEEDS - 1319001 - not really a feed - auto page generated by wordpress
- Latest news | Royal Society - 1321190 - malformed feed, only one item in the feed as well. not updated on vendor side since summer
- Proper Deals - 1322436 - Site appears abandoned. Feed points to a parking site, domain it self still active but no real data
- Royal Society Latest News - 1321191 - 1 item posted in a long time
- Gizmodo - 1014878 - dupe use recommended (which is also broke)
- Gizmodo - 1322173 - dupe use recommended ^
- Gizmodo - 1321360 - dupe use recommended ^^
- How-To-Geek - dupe outdated RSS
- Daily Cup of Tech - 1001754 -web hosting account account suspended.
- Demogirl - 1008556 - domain no longer available
- Above Top Secret - (four of them) - Not Tech but if you had them before you'll have them now.
- Linux today - fixed xml feed to the site one instead of a netscape one (i know right?)