Development site currently at: DEV.DAILYROTATION.COM
Check out the video updates at LBRY.TV
pretty important news from me. -UPDATED
as much as i hate chromium i'm now using Brave as my primary browser. so far so good
I havn't gotten into custimization yet but i'm hoping i can move tool bars around
also switched passwd manager to bitwarden. Much easier than keypass.
I'm working on dropping firefox. Mozilla's latest shit-can-ary has about pushed me over.
I was using TOR browser (based on Mozilla) but the network is to slow for most things not text. (videos and stuff)
side note: if you are using a TOR network, down watch videos/movies and such. you are just pissing everyone off.
I'm looking at librewolf and brave. I can't stand chromium, but.. i've been using Edge for testing DR simply cause it
is the chromium browser i had installed.. anyway .. i wasn't turned off by it one bit. but i don't like microsoft.. so...
I'm trying brave again. I didn't like it much before.. then they got caught having issues with not really being private...
At that time FireFox with uBlock was JUST AS good if not better than brave. Also Brave is a bigger name than LibreWolf.
As they say.. 'hold my beer'.
Back to the notes...
put dev behind a reverse proxy. Working on doing this through all my sites.
still slowly migrating and consolidating sites and switching all to NGINX
Totally dropping OVH hosting. They have no interest in protecting their IP's.
Thankfully i havn't put much there mostly scripts.
I made a haveibeenpwnd script that checks multiple address based on a text file. May make it public.
on DR and Shellcity or somewhere lol.
(almost) no site stuff done. going to take a vacation next month already planing on taking a few hours.
DEV Only needs contact, search and add site. Sortfeeds works but doesn't auto update.. not sure why. probably a logic err.
to work on the site itself. Moving sties i can be interupted with out a problem.. scripting not so much.
with in a few hours i should be able to get these working fairly "quick"
last thing is to add the click tracking for top feeds.
better err checking seems to be working. Still got a couple tweeks to do for insert checking but..
errs there are rare with the rewrite.
Sort feature is working. had a logic err that was on my last nerve last night.. and it was taking
loans on the next nerve... found it.. works. but now AJAX is making me mad. it updates okay but seems
to not have all the css or i forgot to add it in the ajax. will come back to that tomorrow.
also another logic error yesterday... this time in the nginx config... just had } and header swapped..
moved it and it works...
purpose of the header is to disable Googles new federated tracking. If you do not know about it learn about
it now. the TL;DR is that Google is disabling 3rd party cookie tracking (yay!) but replacing it with
federated learning to track you instead which IMO is more jack-off-ary. seriosly avoid google where you can.
(and apple, and microsoft, and facebook, and twitter, and.....)
Anyway, this change is to make sure DR will not be involved in that FLOC tracking.
on new dev server: better error checking on the db. I found if it go back and forth between front end
and back end I stay more interested.
better error checking will man faster updates and a better helicopter view of what might be broken.
Once i verify this works i'll make it 'live' on the dev server.
moving the test site to a different server with some packages that do not play nice with the previous site.
This works out since i want to consolodate all my boxes in to three instead of what is now 7. but the bad part is
that the old ui may not be compatable, though i'm still looking into it.
new site dev site https://dev.dailyrotation.com/ no updates are running on it yet. but 'generally' the same as this
server other than its running nginx rather than apache.
playing with sorting in the new ui. found the problem i was having and it pretty much poops on the rest of the layout.
going to change the way users change sorting to yet another modal and people can drag up and down from there.
Pretty sure this is part of the reason the orginal UI is setup they way it is.
Not sure what to call the option yet or to nest the option in another modal.
'think' i found a provider for the contact form so i'll be working on that this week.
Adding sites will be ready probably by the end of the week. I was givin more serious thought on how to do this
and laid it out in my mind. once i get to it should go 'quick'
another one i was going through in my mind on the way to work, search
think i know how i'm going to do the display headlines.
Think i have a 'good' idea how to add feeds from the search.
Add feeds in general (out side of search) will probably be linked to the search function as well.
okay so the TL;DR. Adding/removing feeds from your personal list is first. Search second since it will be related
sorting feeds and then contact form.
after that should be 'done' other than adding things like user selectable CSS.
while waiting for some PC's to image... Minor changes.
changed menu system from straight buttons to nav. seems like i can get away with out a hamburger for now.
Started the process of consolidating servers. going to have a total migration to nginx. apache is nice and awesome but nginx feels more light weight and 'easier' to use
almost all of my projects will be able to fit on one box. easily. leaving two cheap vms for staging and other small projects that don't need web type access.
This will also make it easier for DR development as all the old stuff won't be transfering over but i may still keep an old.dailyrotation.com for legacy sake undecided.
Dealing with vendors all day.. man what a pain....
Log in seems to fully work on the new UI. As of right now cookie is stored and wille expire in 30days OR until you hit logout.
Privacy menu added to the new UI.
Changed the feed width to about double on most screen sizes. Need to rewrite the menu so it scales correctly
Just easy stuff while waiting.
Ron pointd out over the weekend that logins where not carrying over. Seems situational. I increased the session time out to two days seems okay from myside.
I'm slowly rewriting the login functions for the new UI.
done with the verification
done with pulling data.
I 'should' be finished with applying settings tonight if things work out.
as of right now the only settings that work are the feeds. the New Tab/Window setting i think will be default.
Again - Anonymous users are going away. too much to deal with and i never liked the idea.
Since no personal information is asked just create a dummy account and don't forget the password.
IMPORTANT the 'old' (currnt) site will be going away. there are reasons for this.. top of it.. it is a pain to manage.
I was talking with some creators the past week or so. Despite the absolute horrible and inconsistant
youtube/google policies.. they are the best 'main' platform while also using others. Not expecting
much from it.. just another outlet.
Orginal feeds readded. TL:DR: if it was working before the move, its back.
still looks like about a third of them are not valid any longer +/-
do to the way the new upload script works somethings may not update properly or be delayed.
I'll be working on that the next day or two. Problem is the update script pulls feeds that haven't
been updated in a time frame and updates them.. but if a feed fails it doesn't reset the count.
so the same failed feed could be pulled everytime it updates taking a slot for what maybe a valid feed.
Going to add a flag in the database to stop attempting if it isn't valid (according to the script)
then everyso often i can check it, see if it is valid and if not remove it.
also seeing some click manipulation for the topClicks. Two in particular have vastly inflated
amount of clicks in a short amount of time. Sure maybe it is just popular, but there is some sketchy
evidence too. I see a few adjustments I can make that should determine that. In the mean time...
I modified the count just by eyeballing what would be a more reasonable number, however i made they
are in the same position in the list.
Note: about 20k clicks over the past two weeks or so. much more than i would have thought for a
site that was down for as long as it was...
Along those lines i do see an unrelated issue i missed when writing it. this is just a minor thing
that doesn't effect anything, just now that i see it i cannot unsee it.
Login is coming along nicely but i'm going to have to get some work to anonymize it better.
the less data i have on you the better :-)
Thinking about themes like what is done on the current site. light, dark, etc. doable for sure
Something on the back burner but here so I can remember.
Column sizes. going to have to tweek it some more. Bootstrap is reading 16x10 resolution
as large. which i would consider medium. six columns looks great at 2/4k resolution
but below 1080 not so much. i'll either cap it to four columns or override BS defaults
focusing more on the new UI. got most of the main functions done just got to do some
of the other functions again... add/search along with login and contact.
Contact: not sure which service to use yet. I'm not setting up my own mail server for this.
I also need one for shellcity, piratesperspective, webmasterz and a couple others i'm working on.
webmasterz isn't live again.. but will be my portfolio page rather than the old developer news it was in the late 90's
Headlines will be quick
Feeds will be quick
Adding in an option to put feed into user list won't take long but more logic.
not sure on a UI for this yet. I got ideas but not really sure.
this is one of the easier things i have been not doing.. i don't know why.
I'm running into some problems here. I should be able to get it working but lots of
other behind the things need tweaked.
For those of us who don't like cookies (even my own) I think i'll set the session time
to 24 hours so only one login per day is needed.
for those of you that like cookies.. whatever you are weird but i'll still make it an option.
What I am doing today/night that will effect current live i'm putting all of the old headlines back
why? i noticed some of them where blocked.. now they're not. we'll give it another go i think.
check out the new desgin: HERE (keeping in mind alot of things don't work)
do you like the old or new better? like the new one but not the colors?
Again, thank you for your donations!
Current live site: going to add ALL the previous feeds back into the updates. I'll probably do that this weekend.
it requires some new logic to do.
Focusing on new layout.
Feed display finished
top feeds finished
site is "fully" responsive. Columns adjust to your screen size.
at 1680x1650 + resolution 6 columns.
MOST phones in portrate mode one column. many phones in landscape 3 columns.
I like the facebook color style. Going to tweek it some yet but might go another route with the scheme once finished change will be easy.
changed up the links to be modals instead of in the page directly. none work yet.
Still need to write and intergrate user/search/contact forms.
One of the planned feature additions i may have mentioned a long time ago will put too much of a load on me as resources required would go up if people use it.
I may make this a donation feature of some sort,
I skipped over the whole adding and went to making a new UI.
Sneak peak: https://www.dailyrotation.com/bs/ [ not a direct link to keep robots off of it ]
The feeds are just random ones i pulled out... the 'nothing here' are feeds that do not exist currently.
using facebook color scheme basically. say what you will about facebook (the bastards) their initial design was pretty great.
Repopulation seems good. I'll have to write another function to remove old old items.
about 50ish feeds still broken not the fault of the script.
adding feeds will hopefully be fixed this week.
'should be' simple. just finishing up another project plus work has been fairly busy.
Once i get that finished and some other things out of the way i'm going to work on a more modern
Interface while keeping it simple like the current.
Finally decided on a way to 'fix' the feed ordering. In the end i just went with nuking them and then
re-adding them. was just easier than going through and updating all the others with the new order value.
They are repopulating now.
I wrote a search function, but i'm not sure how i'm going to get it to display as of yet.
** update ** search works, not pretty but finds the headlines just fine. Removed all the latest for now.
You may see multiple results for many of the feeds. this is because there are quite a few duplicate feeds and i'm not going to take the time to weed those out since many users have either or.
There is also some instances where multiple feeds use the exact same information >.<
seems like i also need to make another update to remove old feeds. I think i'm going to remove any items older than 6 months? thoughts let me know.
Added last updated to the feeds. some math wonky, just leaving it with the date.. wonky math has to do with timezones, its good nuff as is to tell if feed hasn't been updated in a long while... (every feed "should" update daily at minimum).. moving on....
These feeds ordering backwards is driving me nutty. Even worse now that some feeds changed date format in recent months (engadget i'm looking at you)
Its not a bad thing they did this... just hurts my head. I have to fix it.
going to add a 'Last updated' on to each feed. Probably this weekend.
So... i added my click tracking right? right. so here is basicly what happens. It downloads the feeds and compare's them to what is already there.
It is comparing 5-50 feeds into the current list. well... here is where i messed up. I compared those downloaded feeds against everything in the databse
all 75k+ so.. yeah it took awhile to just do a few feeds. Since the job runs every 30 minutes... it got mad. changed the function to only compare against
that feed and zoom.
I'm working on another project, a little personal daily task tracker. Not hard just i hate making UI's. Probably should just look for one that is already
made and not reinvent the wheel.
again search is next on my list, easy but intergrating it into the UI will be a pain. then the adding of sites. Easy just doing a simple ui for it...
If you havn't noticed I hate doing ui Especially on this site since i want to keep the look the same and its hard for me to pick out code. So i keep
putting it off lol.
Thank you for all the donations! I am overwhelmed. Might have to setup a patreon acocunt (or similar) -- NOTE TO SELF -- add crypto as an option.
Found another feed like downloadcrew in haveibeenpwnd. I will have to look further into this.
Oliver pointed out an oversite.. well two..
I forgot to account for 0 headlines - corrected
I made a type-o in the updater/topheadlines functions that killed the updates. -- corrected
have to add in the newtab option yet but other than that everything is done EXCEPT for search/all the latest and add
I keep saying this but keep getting pushed back on other things that are broke
my prioritys are in that order. search then all the latest seem like they will be basicly the same only with a pre-defined search topic.
Search i'll have option to search headlines or feeds.
Planned feature update (after things are working)
Random Recommended feeds. Not sure i like this idea because of how many none sci/tech feeds their are but think i'll change the title with out the Tech and Science label.
There are a few I will keep in the recommended.
Random default headlines
There are a few I will keep in the recommended
Both to change weekly (or there bouts)
Those kept in these will be my sites and any donater sites if they would like with the assumption that they are not overly crazy sites.
IF YOU RECENTLY DONATED and would like a site to stay in the default feeds for a bit. Let me know.
Found another bug. This is due to different sorting different RSS feed formats which is causing issues with sorting
Example is Download Crew
This means I need to make another change to the updater
This isnt' a bug its sites doing their rss feeds back-asswards.
Sorry fixing it for a few feeds is not worth the headache for their mistakes.
I even considered rewriting the main site just because of this... I found a can of nope in my backpack and cracked it open.
found a bug in the top headlines. really its a bug in the updater.
updater creates a new item id each run for a site. top is based on ID.
think i'll modify top instead of updater simply less to dig through.
Modified updater lol.
Clearing top headlines shortly. Will monitor
UPDATE: New tab targeting broken. I think I know why. i'll come back to "shortly"
Takeing a break from work.. at work to calm my nerves lol
If the main page breaks.. sorry i'll restore it shortly.
Top Headlines now working. Top 10 only for non-logged in users
Since this is a "new" thing... you will mainly see my clicks i used for testing, once things get going this should look more normal
I do have to clean up the spam check a bit to make it more anon... i'll come back to that tonight.
after that i will work on 'all the latest'. I'm really not sure what that is supposed to mean yet.
Privacy: your information is stored in a cookie and on our side. We (I) will not let it into the wild on purpose.
Non-account users have a default user account assigned to them and that is stored in a cookie and on our side for a short time.
your IP is logged in our anti-spam system when you click headlines.
No information besides time of click, ip, and id of the clicked link is stored.
If you do not click for a couple hours this is automatically cleared.
I will be working on anonymizing this more.
I strongly encourage using a generic user/passwd that is not tied to any other online account.
If your twitter is @jane_doe I suggest not useing jane_doe for account name. Also a fairly generic passwd.
Why do i say this? I don't want to be responsible for your account security.
There is NO data stored that could cause you harm but if you use the same user/pass combo on other sites
then you may run into a problem should something happen.
You shouldn't be reusing passwrds anyway. get a password manager. (keypass for example)
Do what you want don't break it.
Try to keep things sci/tech related.
I reserve the right to block or delete anyone or anything for any reason.
Only reasons i can think of right now are
attacks or abuse of API's (i may create later)
an flood of non-tech/science feeds
though i'll probably just let it be as long as it is semi-balanced
Bob created this site a couple decades ago.
Bob, Gerry and I did a bunch of work on it behind the curtain, code, ui, feeds, server, etc...
Bob retired from web stuff, joe took over and broke EVERYTHING slowly bringing back stuff.
Where donations and ad monies go to. All will go to my web projects unless it pays off a year of them (yeah right)
after that.. i'm not sure yet. I don't think that will happen.. will cross that bridge if/when we get there
I need one, people are missing the [ at ].
Leaving the AddThis for now. I think I might work on my own.. but we'll see.
Going to attempt to create a video on all of this tonight
Cleaned up junk here that no longer applies since I changed directions
Got top clicks and spam checking working last night.
Most of my time since is trying to decipher how it is currently intergrated into the front end.
There are just so many "includes" in the old code, its just all over the place.
Making plans to move servers again.. wha? i know right. Once i get this stuff working then moving is as simple as copying files, running 3 line script and pointing dns to the new server
goal is to consolodate my five virtual machines into two or three virtual machines, two of which being identical to each other for backup and the third for reverse proxy.
reverse proxy pretty much links the two each other. All three on different web hosts. currently using linode, digital ocean and ovh for my virual domains (not counting my home lab)
eek! it looks like a script runs an update for the top headlines and then writes flat files >.< this is going to suck lol.
Okay top template almost done but got parts for my lab today, don't know if i'll get back to this for a few days.
I WILL BE REMOVING THE [sh] LINK! Add this is a pretty shady entity with data... at some point i'll look for direct share links but some api's have been removed or changed since the last time i went to use them (in 2012 or so)
Stuff i worked on last night. Top list. rewriting the outboud clicks functions.
I'm going to be adding this to the main page, things may break. if the link says "out.php" and it doesn't work.. that means i failed and probably already fixing it, but still let me know
'hold my beer' as they say.
Doing first major OS patch with the "working" site. "Hold my beer" so the saying goes....
Finally updater working.
I got some personal projects i'm working on along with my 9-5 job so in the near term updates will be limited.
this week i hope to be closer to the top XX working. and updating some links previously going to Bob.
At the end of all of this, besides the actual UI... about 90% of the site will have been re-written into much leaner and updated code.
In a braindead sleep depraved moment i got an idea and it worked.. but i disable the updater and just realized tonight i never started it again.... .. its starting ... wait for it...
Feeds and updating are working. found three that have never updated. Boradband reports, space daily and DV hardware. there are a couple that i don't think are updating either
Next project getting the add feed to work. should be easy. the function exists in a include that no longer exists.
after that top xxx. I believe these where done with flat files before but think i'm going to create a new table for them for easyier searching
I honestly don't know what this did before but shouldn't be to hard once everything gets added to the database. this and add feed may not work with the current page format.
It may not happen this week. I'm working on another personal project. Never learned to do it so i'm learning revers_proxy. I never used docker because of port numbers that are blocked. this takes care of that for me.
Along with that, I setup room in Matrix using Element. for those that don't know Element is a more secure and Private messaging platfom similar to discord just private. no email address required.
New video should be going up today going over this weeks progress.
For those wondering. The update code was reduced to about 200 lines including comments.
added a bit of error checking last night and logging. Need to clean that up some to get easier to read info.
quite a few feeds are still broke but not the fault of DR. either RSS for the site no longer exists or moved. Checking ones that pop out at me and updating urls where possible.
last night the update completly failed but had to sleep. error is i was returning in the wrong spot. quick 1 minute fix
re-running an update to all feeds now.
Decriptions now work on the feeds for almost all. some are failing on insert. Need to put a check on that to blank out the description if error.
once that is done i'm going to work on the top feeds. after a quick look I think i'm going to rewrite it for more modern code. It might be as simple as a permissions err.
Okay all set with what works which seems to be 90% or more of the feeds. going to pick through the remainder some other time.
yup, i'm rewriting the top feeds. I just don't know perl enough to make semi-major changes. I think it threw alot of errors before like it is now so maybe it is working and just won't really update until enough traffic comes through.
Either way it needs rewritten. I found some stuff in here from before that isn't acceptable in 2021.. 2010 sure.
But no matter what it will perform better :-)
running one last update before i put it to auto updateing. Note it might be a little wierd for a bit. Not all feeds update per fucntion but every feed "should" update every 3hrs or so.
I'm going to work on other projects for a few days, but if there is something specific or feed you would like to see let me know and i can come back to it.
set to run at lunch, WORKING GREAT!
Still need to fix error checking massivly but so far so good.
Many more feeds that previously didn't load are not loading.
ALL OF THE DEFAULT PAGE now works.
ALL but a handfull of the recommended works. A few i know why and may or may not be removed. TBD
THe updater IS slower but i am also letting it parse more so.. +/- Still updates 1k in under 30minutes. so..
I'll work on it some more tonight
WHOOPS! I aranged all the feeds back-ass-wards hahahah should be fixed the next time I run an update
updating this late as i'm waiting on stuff to finish at the office
Updated a bunch of feeds and started flaggin feeds that don't seem to exist any more.
This has got me wondering if some of the old feeds i removed might still be valid. Will revisit this later (i still have them as i thought this might possibly happen)
I'm on to v4.0.3 of the updater. again found what i think is a better way.
As they say, "Hold my beer"
Guess what I did yesterday and a bit today? wll did a vid on the lbry.tv page and rewrote the entire update class
why? speed mostly. the previous improvments got me some ideas.. and also made error checking easier.
the class is much smaller but the calling script is a bit bigger. I suppose i could put that in the class too but that would get overly complicated.
hahah! that didn't work think i know why....
leaveing old updater for now.
Any thoughts about having a community chat group? Let me know what you think!
All Negative, I'm free tomorrow!
Netcraft had two near identical entries except for the description and id. one worked and one didn't.
"MOST" fo the recommended feeds (select able at the bottom) work
only a crap ton more to fix. i have a feeling they're similar problems.
Tests hopefully return tonight (cross fingers)
found a way to speed up the updating which inturn sped up the front page. Got a few other ideas to speed up/reduce the amount of queries. Over 6k queries an hour!
changed the update flow again every 30 minutes it updates a portion. I need to tinker with this a bit to make sure every feed gets hit but it will get there.
setup an internal stats setup. If it works half way decently going to remove Google. Initial look shows way more people looking at this site daily than i realized. Sorry i'll try to get stuff done faster! as always if you find something you specifically want working mail me and i'll put it on my list!
Hopefully will get the error checking done today to find out why those feeds are not updating.
cleaned up the queries MUCH better now.
I still keep bouncing around the some feeds not updating thing. ran into something that might be doing it... but.. i'm going to bounce around that again for the night ha!
One tested Negative but need to do more tests! man what a pain.
Other than the ongoing issue with some of the feeds they seem to be working, updating every 4 hours now
Not been doing Much. Family still on the COVID watch and this hasn't been a priority though still keeping thoughts on it.
Thanks to Ed K. for letting me know about the 404 error. forcing ssl messed up my rewrite.
Speaking of, Addled ssl requirement. Google Analitics, add this, etc.. cause it to be n0t 100% becuase of external files.
Will probably remove these, privacy issues. ::middle_finger:: google will be looking for another non-evasive statistic service
Some feeds with valid feeds are still not updating. going to create better error checking and logging for these and go from there.
Next priority is the top lists. I have no idea how this really works... yet
While I havn't made much site progress i've made some other external progress. More to come.
Waiting on another COVID test to come through. Third time i've been isolated >.<
During this time i found what i think is emptying the database. going to start up the cronjob for updating again.
installing updates for the os, hopeing it doesn't break shit. Thankfully I have backups.
Over the weekend all the feeds broke again. I cannot find a reason for it. there is nothing in the old site set to update or delete the table the feed items are in
only two things i can think of is something in my logic is deleting them. I disabled the delete function. seems to have worked for a day.
other thing i can think of, I was borderline with vm resources and it is just breaking. I upgraded the VM.
Need to get the number of queries down and also disc reads. However i believe this will also break the old code.
Scarlet Vision comes out on Diz+ today! got to binge mando to get to it.
In pushing to update the default feeds today.
Trying to avoid politics on recommended or default. It amazes me how Politices has infeltrated Tech.
It gets even worse when you realize its is so one sided. The truth doesn't matter any more... /sigh.. anyway thats my statement on that.. expect very little of it Geek before spreading the word of liberty here ;-)
Once i get the default done I think all i really need to do is fix search and top lists.
For the feeds.. since sooo many of your feeds are gone I am going to put ALL feeds in the recommended section for a month or so. YOu may see some politices there that you don't agree with... suck it up butter cup. Half of america doesn't agree with you and you know what? That is okay different opinions does not make someone bad.... oh shit.. thats another statement.. i lied sorry.
Thanks to everyone stick it out. I'm getting there.
Default done, need to remove one or two it seems. monitoring
Just realized how long its been since i updated shell city. I really need to remember.
found some feeds not loading did some error checking.. broke them all. working again but still those handful not.
other wise seems to be okay. might mess around with that more over the weekend.
Did add some more err checking but its not picking up any errors... which.. i don't understand, there are errors >.< maybe the error is me
going to add some vm backups and then try to update the OS again.. maybe tonight but probably tomorrow X your fingers.
Yesterday found an error in the updater that was not inserting valid feeds. I got an idea, and then annother and then..... ..
I re wrote the updater this morning. it faster, more efficient and more forgiving... unless the feed it self is complete garbage.
for example daily cup of tech doesn't exist now and the suspension page really made it mad. the right error checking shoudl fix this... later.
ran an update to all remaining 900ish feeds it seems almost all worked properly.
Updated the recommended list. there are still a bunch of dupe's but getting there.
tonight or tomorrow updating the default feeds.
Got an account? let me know how your feed is looking send me screen shots or whatever If i took out one of yours let me know we'll see if we can get it back!
Busy day, nothing done that would be visible. only small changes that do not effect the main page.
Noticed some fatal error issues last night. old code popping up again. I think this might happen from time to time until i get a more complete rewrite done. (maybe never lol)
Some feeds don't display at times but works on the next. I think this is a database issue. Going to check the back end.
Ajax not working on the fly, initial testing, loging out and back in seems to correct this. might be a cookie issue.
Didn't get to remove/fix feeds last night. If nothing crazy happens to day i'll pick through them.
Investigating why some valid feeds are not updating properly in between work stuff
I think I figured out why those feeds do not appear to be updating. Mixed methods of updating/listing appear to be causing the problems on some feeds. This is a change i was planning on doing later anyway but not for this reason. I hope it doesn't break what was working lol.
After removing many that simply did not work i'm going to create a list of ones i'm removing that don't fit. this list will be at the bottom of this document.
Ever have one of those OH DUH! moments. Yeah yesterday, hit that.
Found out why my updater worked on the rewrite but not on the old page. I never disabled the old page so it was updating twice and since the old updater doesn't work on this server......
MANY feeds include media and HTML in their feeds. This is 'okay' normally but doesn't work here. Need to sanitize.
removed about another 500 feeds that where not working at all. If you found one missing that you want. you should be able to readd it useing the add link.
I removed nearly all the recommended feeds. I'll be readding a bunch. I'm guessing probably around 50 or so.
Once i get the inserts working and get recommended going I'll be working on the top list(s)
There are around 1k feeds left in the database.
Some updated policies I will generally follow and plans (my site i'll do what i want)
I will only be adding Sci/Tech too it recommended I find interesting. If you have a suggestion let me know. Sci/Tech only. YOu can add other feeds but i will not put them on the recommended.
Stuff like EFF, Internet Censorship, Net Neutrality, sec. 230 and such is good as it is at least semi related even if more political than Tech as long as they are not generally political feeds. (like huffpo tech feed okay, huffpo main feed not)
So add them, if you think you want them on recommended good ask.
I may add links to sites that maybe political in general but to articles that relate to tech and lobby for things like Net Neutrality.
DailyRotation twitter is not me. It is Bob's but feel free to talk to him, cool guy :-)
I quit most social media so I have no social media contacts. Use the contact link to get a hold of me.
Some plans on other "non mainstream" social sites but not something terribly important.
Part of the reason for not using the "big guys" is to add some variety to other networks rather than just political stuff
now that other things are working, i found some errors on inserts. Link field too small character encoding. Previous check for encoding is depricated and no longer works. May have beena problem on the old setup but i didn't notice since my involvement was just to attempt to keep the old VM up.
for some reason the link titles are coming up blank on a good portion of the links.. still.. moving forward instead of backward +1
THink i generally fixed the blank links
a handful of feeds have valid feeds yet still are not updating.
going to ride it out a while to see what happens.
Changing my auto updater task. I'll be starting it up in about an hour. It will be configured to make sure they update at least twice a day.
In the mean time i'm going to update my recommended list and update the default feeds sites that "should" be working should have this mostly done tonight.
magpi is still trying to get in there, removed all its functions (from what i can find) and removed it. Removed magpi from other modules that i could find. I have a feeling it will pop up again but one problem at a time.
If there is an RSS feed Missing that you had before let me know.
If nothing dreadful happens over night, i'm going to enable SSL. Look at optimizing the database better and consider upgrading the vm or offloading the DB to another server. DB speed isn't bad for just a few visits at a time now that i'm not updating 1k feeds at a time.
the default page is now working. HOWEVER i'm clearing the headline database and going to update all the feeds tonight.... i hope.
the TL;DR of the MAIN trouble i'm having. Old code is from 1914 and badly needs updated. New code just doesn't want to work with the layout that i don't want to change.
the good news is, the worst part with the old is the background updater.. and that is one of the things the new code is doing the best at.
I have it updating all bazzillion feeds now. Will see the front end picks them up. Site will be god aweful slow today (not that it really matters).
There is still work to be done for sure. Some of the formatting on the feeds isn't working properly. likely just stripping slashes and such.
need to clean out the feeds too. I already got about 1000 out that havn't updated in a year before the site transfer.
I "fixed" the updater to make the bad feed checker work a bit more not bad. so once it finds all the broken feeds i'll pull them out.
today i learned that the only feeds showing up in the checkboxes are the recommended ones.. there are hundreds (if not thousands) more in the data base. I mean just eyeballing it should have noticed but i didn't think about it until i started looking at the user functions.
One of the perks to doing all of this. I'm seeing a bunch of feeds for sites I have forgotten about over the years.
I'm seriously considering removing all non-tech/science stuff out as i find it. I mean i added babalon bee to the recommended.. but that was just testing. There are many that fit this bill.
I also have a plan, if the user base should return after this whole mess.... of making a list of all the feeds for users to vote on to make recommended.
Also, thoughts suggestions or just a note so i know that my time isn't being totally wasted. (i'd do it anyway though)
there are a lot of id's from recomended feeds that just don't work. I need to change my error checking script.. other wise seems much better. i also noticed that the ajax for the removal and moving of items isn't quite right. If an item is in the first slot it cannot be removed. if you move it to another slot you can remove it. might be the whole array starts at 0
My cares are gone today. scrapping what i've been working on and starting over. I missed something and i cannot find it now.
Also looks like my updater broke. 'should' be an easy fix.
figured out my problem... but my time upcoming is going to be limited. Instead of upgrading the code right now i'm going to attempt to just repopulate the database and go with the old.
Got feeds "mostly" formatted. still some minor issues, but the ajax dragging seems to be working. but don't have it configured to save it yet.
Changed Default feeds to be random 25 feeds instead of static. Gives more varity. will probably change this to be 25 random feeds for the day to reduce the query from updating every visit to query once a day.
readded a bunch of code to get the overall page format back to normal. Will have to pick through to see what I was missing later.
my current re-write of the back end got bloated to 170 lines with comments. I'm projecting about 250lines for feed updates and display and about another 1-200 for user functions.
seems like every change i make today boogers up the feed formatting.
every change is messing stuff up because I found another logic error on my part. if else for while son of a bitch lol. Once i find it things should be falling into place again as i know the other issues now.... .. i hope
Over the weekend having lots of logic errors (on my part) got no where useful
Logic starting to work for me. Got the feeds to display. Working on formatting to previous layout
formatting getting there, but seems like in the process I busted the previous CSS which might be trickling down so now i need to walk back up...
Added feed updates to cron.
working on getting feeds formatted for main page. going horribly
pretty busy day, minor things only.
running updates in the background. These updates "may" show on the main page using the old code. This will be replaced so any changes made won't be reflected.
In the processes of the updates it is watching for broken feed links and some malformed documents. also remnents from an injection attack years ago.
Guestimate of feeds to be removed, i would say around 50% which would still leave around 1000 feeds. Many of which may still be broken.
Most feeds now updated will now run around 50 random feeds every 25 minutes until live. Once live comes i'll change this to 50 every 10 minutes.
This should give the feeds better 'feel' and update more consistantly.
some progress on the userfacing side yesterday and today. will update this info later on.
12/29/2020: Latenight/early morning -> cont
feed display testing complete. Next adding a single feed to current site.
full back end functions for getting and displaying items less than 130 lines including comments! Actual calling of functions will be a bit more but not much
Domains fully transfered. Big thanks to Bob and Gerry!
DB Query update done. Will update them to "newer" implementations at a later time
Feed update done. Just need to work on the logic for the timed tasks there are MANY feeds a good portion of which no longer valid
Created a function along with the feed update to identify dead or malformed feeds.
Tables work with the new connector flawlessly. almost don't want to update the connector.
backend nearly "finished" for the feed updating
cleaning up my code to remove testing. Less than 150lines.. could be less but for another time
Feed stories cleared, starting fresh. when live not all feeds will be populated
found a more efficent way to do error checking on feeds. implemented.
feed objects breaking becuase of the error checking and malformed RSS feeds. Still a minor issue, may drop description field to remove these problems for the time being.
"assuming" the feed updating works, will work on adding it to the main page.
Error checking to find dead feeds, done. Will need to run a complete check as live comes closer
All feed requests are now using local code instead of third party.
Feeds Fixed (but may not be updating still)
- Techbargins - 1000554 - feed no longer tech bargins. Couldn't find an updated feed to the electronics/computers
- QRPSpots - 1322394 - RSS feed no longer updating from vendor. no items returned
- The Tremblin Earth - 1322091 - No updates since 2018 RSS malformed
- Wired News: Top Stories - 1000428 - Dupe
- Comments on: WIRED RSS FEEDS - 1319001 - not really a feed - auto page generated by wordpress
- Latest news | Royal Society - 1321190 - malformed feed, only one item in the feed as well. not updated on vendor side since summer
- Proper Deals - 1322436 - Site appears abandoned. Feed points to a parking site, domain it self still active but no real data
- Royal Society Latest News - 1321191 - 1 item posted in a long time
- Gizmodo - 1014878 - dupe use recommended (which is also broke)
- Gizmodo - 1322173 - dupe use recommended ^
- Gizmodo - 1321360 - dupe use recommended ^^
- How-To-Geek - dupe outdated RSS
- Daily Cup of Tech - 1001754 -web hosting account account suspended.
- Demogirl - 1008556 - domain no longer available
- Above Top Secret - (four of them) - Not Tech but if you had them before you'll have them now.
- Linux today - fixed xml feed to the site one instead of a netscape one (i know right?)