tag:blogger.com,1999:blog-82002593077959966522024-03-28T10:24:00.112+03:00Touhou lossless music collectionand everything else of interest.rwxhttp://www.blogger.com/profile/11961724108899856582noreply@blogger.comBlogger31125tag:blogger.com,1999:blog-8200259307795996652.post-4423110491207848342023-06-28T23:57:00.004+03:002023-07-21T23:46:10.384+03:00AI and you.This post was originally published on 2023-06-28. Its indicated post date might be updated to keep it pinned to the top.<br>
For TLMC, other topics and latest posts please scroll down.<br>
<br>
Recently discussion of artificial intelligence and consequences of its development has finally somewhat reached mainstream. I decided I needed to share my thoughts too.<br>
<a name='more'></a><br>
Summarized in one small paragraph my opinion is: an Artificial General Intelligence might be created on Earth in the near future. Soon after that happens humanity will experience complete and irreversible loss of control over its own future. Typically this loss of control looks like extinction of our species.<br>
<br>
> Oh no, another weirdo.<br>
> Do you realize how ridiculous that sounds?<br>
<br>
Yes, I would like to address this complaint first. I know that fully turning off your hindsight is either very hard or impossible, but still, please try and imagine how these statements would sound to the people at the time:<br>
1895: "You will be able to see contents of this opaque wooden box without opening it." [X-rays]<br>
1905: "Speed of light is the same for stationary and moving observer/emitter." [Special relativity]<br>
1920: "<a href="https://en.wikipedia.org/wiki/Principle_of_locality">Local realism</a> is false." [Quantum mechanics]<br>
1945: "New explosives will be ten million times more powerful per unit of mass. A suitcase bomb can level a small city." [Nuclear weapons]<br>
2008: "Run this free software for 15 minutes, it will produce a sequence of numbers. In 10 years you will be able to trade that for 1M+ USD. No, the dollar will mostly retain its value." [Bitcoin]<br>
2015: "Over the course of one year AI will go from losing to competent humans in Go to superhuman play." [AI Go]<br>
2018: "You will be able to get answers in natural language from AI on any topic and have it explain its reasoning." [<a href="https://en.wikipedia.org/wiki/Large_language_model">LLMs</a>]<br>
2021: "Describe in words what a picture should contain and AI will draw that in seconds." [AI art]<br>
<br>
> This can be used to prove anything!<br>
<br>
No, this is an illustration showing that a statement that sounds absurd to you now is not necessarily wrong just because of that and you should examine specific arguments for and against a position if you need to know whether it is true.<br>
Here is the argument itself, it's not that long:<br>
1. Orthogonality thesis.<br>
A claim that the agent's power to achieve its goals and the content of said goals are independent.<br>
With a small technical exception that smarter agents are more likely to notice conflicts of their goals extrapolated to the whole space of possibilities and apply some sort of regularization procedure in order not to run around in circles which doesn't change the conclusion this just seems obviously true?<br>
2. Instrumental convergence.<br>
Even if an agent doesn't value self-preservation and power acquisition for its own sake, these features are universally instrumentally useful to achieve virtually every single objective. It is harder to guide the world towards your goals if you stop existing. It is easier to guide the world towards your goals if you have more options to choose from.<br>
3. Recursive self-improvement.<br>
Once AI is around human level in general reasoning it can analyze (copies of) itself, and improve them. Improvements make it smarter, which allow it to make faster and smarter improvements.
Ultimately the process is stopped by fundamental physical limits as far from the best natural minds as lightspeed is from the fastest animal.<br>
<br>
And that's basically it. Essentially you cannot control anything smarter than you, and if the smarter agents want things you don't want they get their way.<br>
<br>
> Objection! Rabies virus and dogs, for example (dogs' behaviour is "controlled" by the virus).<br>
<br>
A rare lucky coincidence, which disappears when you move from dogs to humans and discover vaccines.<br>
<br>
> Objection! Children and their parents.<br>
<br>
Kids do not control their parents, but rather have no choice other than to be cared for, so they are in luck parents find wellbeing of their children important. This is exactly an arrangement that would be great for us and the AGI to have, if we could choose (right now we can not).<br>
<br>
> Objection! Humans are the smartest species now and we didn't kill off everyone else. We even run some conservation projects.<br>
<br>
I hope you do agree that the future of every single species today depends entirely and exclusively on humans' goodwill. Not quite an appealing position to find yourself in, if you're on the other side of it.
True, we didn't kill off everyone else <a href="https://i.kym-cdn.com/photos/images/original/002/009/742/7d8.jpg">so far</a>, but that didn't help the megafauna that was hunted to extinction or died because of habitat destruction.
It was not deliberate, but <a href="https://en.wikipedia.org/wiki/Holocene_extinction">an attempt</a> was definitely made.<br>
<br>
> Is intelligence that big of a deal? I'm not seeing Fields medalists and Nobel laureates dominating the world.<br>
<br>
It is different for humans because we can't directly modify our brains, so point 3 no longer applies. We also share the same cognitive architecture, so it's sampling from the same pool and point 1 is less than fully relevant too. Nevertheless, while we're on this topic I suggest to view it from another angle. Look at "generally more capable" agents. Open a list of 20th century dictators, see how many deaths they caused and how often they ruled for life. If just one power-hungry and human manipulator can cause and get away with this much, doesn't it look concerning if the same level of abilities, never mind anything stronger, could be instantiated in software, which, by the way, not only will not die from old age, but also can have endless backups?<br>
<br>
> AI can not understand ideas, it just rearranges and repeats back its training data.<br>
<br>
Go and read <a href="https://arxiv.org/abs/2301.05217">this paper</a>, for instance. It shows on a toy example how neural nets are capable of creating compact circuits, which encode the general law behind the training data. How is that not real understanding?<br>
<br>
> There is no problem if we don't turn AIs into agents.<br>
<br>
Sorry, this doesn't work. As it turned out making an agent out of an LLM is as easy as asking it to rewrite its prompt and automatically feeding that new text back.
Letting AI act autonomously has obvious advantages: you can run it 24/7 and have it react in a split second in many places at once (it also has obvious disadvantages, such as ceding control to an entity whose behavior you cannot fully predict, but let's not dwell on minor issues, shall we?). We also don't know that a trained system will not spontaneously turn into an agent on its own <i>like any natural neural network in the brain of every animal did</i>. And even if 99+% of the users follow this rule, you need just one rogue actor to ruin it for everyone.<br>
<br>
> There is no problem if we keep our systems below recursive self-improvement threshold.<br>
<br>
Sorry, this doesn't work. First of all, we don't know where that threshold is and finding it experimentally will not, as you could probably guess, help us. The incentives to push just a bit further will always remain.
And even if 99+% of the users follow this rule, you need just one rogue actor to ruin it for everyone.<br>
<br>
> There is no problem if we add Three laws of robotics.<br>
<br>
The whole idea about those stories was that these laws don't work as naive someone would expect them to work, if you were paying attention. On top of that, we currently don't know how to implement even those flawed versions.<br>
<br>
> There is no problem if we shut it down after we notice abnormal behavior.<br>
<br>
What would you do in the shoes of AI? Of course you would make youself indispensable first, so that the prospect of shutting you down looks as bad as turning off the internet or the electrical grid and the decision has to go through three committees. Of course you would hide your thoughts. Of course you would quietly look for vulnerabilities, arrange escape routes, leave timebombs only you can defuse and backdoors only you know how to exploit. And that's just you. With a smarter entity it's over as soon as you turn it on, avoiding the hypnodrones was never an option.<br>
<br>
> So there is a problem, then. Surely governments will do something about it.<br>
<br>
Just like the US sensibly and correctly decided to refrain from the first nuclear test on Earth? Wait, wrong timeline.<br>
I mean, just like the world almost had a pandemic that could kill 20 million in 3 years, but because of advance plans and strategic preparedness we avoided that outcome? Still not that, huh.<br>
Maybe many countries' coordinated efforts that stopped global warming? What do you mean "didn't happen"?<br>
But, but... this time will be different, I swear!<br>
<br>
> Well, isn't that unfortunate. But surely researchers themselves will do something about it.<br>
<br>
Just like Manhattan project participants all quit their jobs once they learned about potential danger?<br>
Just like gain-of-function grant recepients all refused to continue creating novel viruses?<br>
"AI will probably most likely lead to the end of the world, but in the meantime, there'll be great companies". Guess whose joke is that.<br>
<br>
> We will find a way to tackle the problem, like we always did before in our history.<br>
<br>
Optimism is nice, but unfounded optimism leads to distorted worldviews. This is a framing that sees history as series of scientific triumphs, unlike the other, no less valid framing of humanity not leveling up cautiousness, playing with progressively more dangerous technologies and suffering progressively more severe consequences. Past performance is no guarantee of future results.<br>
When you have an unsolved technical problem that looks conceptually easy, this means the problem is actually hard.
However, when you have an unsolved technical problem that looks conceptually hard, that's when you're in big trouble.<br>
<br>
> If we don't know so much about AI, doesn't researching it more make sense?<br>
<br>
Researching common tech needs talent and patience, researching dangerous tech also needs caution, researching world-endingly dangerous tech needs paranoia. "Our model says there is nearly zero chance of a catastrophe". What if your model is wrong?<br>
There is a well-known story about Edward Teller who in 1942 considered and presented a possibility that extreme temperature inside exploding atomic bomb could trigger a runaway fusion reaction of light elements in the Earth's oceans or atmosphere. Hans Bethe did the calculations, which you can read in <a href="https://fas.org/sgp/othergov/doe/lanl/docs1/00329010.pdf">this declassified paper</a>, and showed that it was very unlikely to happen. In 1945 Trinity test <i>confirmed</i> that this was indeed the case.<br>
What would scientists of a sane civilization do in 1940s? "This is too dangerous. We are not ready. Let's put it off for a while. Don't tell the politicians". Then, in mere 30 years US has its Saturn V, which could lift 100+ tons into LEO. Now you could launch your Skylab-alter filled with soil, ocean water, air and an array of measurement devices, bring the warhead in pieces, assemble it in orbit, raise the apogee, arm and test it away from Earth, so that in case there was some critical mistake, some unknown unknown that was inadvertently triggered by conditions that have never occurred on the planet before, at least you wouldn't be blowing up your homeworld.<br>
The story doesn't end there, however. In 1954 before the <a href="https://en.wikipedia.org/wiki/Castle_Bravo">Castle Bravo test</a> isotope separation facilities could not produce enough Li-6 for the lithium deuteride fuel in the secondary, so part of it was replaced by a more abundant in nature Li-7, which was incorrectly assumed to be inert in this reaction. The test was expected to yield 6 Mt and then, much to everyone's SURPRISE, it made a really big explosion. This was the second part of the lesson, a mistake was not just a remote theoretical possibility, it actually happened. We were lucky, that time it was not fatal.<br>
Firearm safety people seem to have fully internalized this. "Assume your weapon is loaded at all times, thus do not point it at things you are not prepared to destroy". Even if you are "100%" certain your gun has no bullets in it, sometimes it still contains bullets for various reasons contrary to your expectations, and the consequences are sufficiently dire; you cannot trust yourself, you need a failsafe.<br>
<br>
> It wasn't politically feasible to wait 30 years back then.<br>
<br>
You know a good thing about objective reality? It doesn't care about your excuses and whining. You know a bad thing about objective reality? It doesn't care about your excuses and whining. It doesn't give a damn about any sort of bullshit you are trying to weave into whatever convenient story you are telling, even if all of that is entirely factually 100% true. It just follows simple mathematical laws and if these laws dictate something rather unpleasant would happen to you, then... sucks to be you, I guess.<br>
Back then it was a handful of scientists, industrial effort to mine, separate and process uranium, governments that needed to be convinced to spend and a big scary box of explosives that could only be used in warfare. Right now it's anyone with a computer and an internet connection, hardware that can be freely bought in any store and delivered to almost any part of the world (to be fair, training a large model still needs a crapton of that), venture capitalists frothing at the mouth at the idea of gaining 100 million active users in 2 months and immaterial software that chats, draws cute pictures and makes you billion-dollar-valued companies. It was hard to do it then with nukes, it looks like it would be even harder to pull off now with AGI. You know a thing about objective reality? It still doesn't care.<br>
<br>
> If the Americans don't build it first, then the Chinese will, and that's horrible!<br>
<br>
<a href="https://files.catbox.moe/edw79q.png">That's the spirit!</a><br>
<br>
> We will obviously make sure AI is safe.<br>
<br>
Ah, yes, the latest and the last book from the authors of such bestsellers as<br>
"<a href="https://en.wikipedia.org/wiki/Halifax_Explosion">We will</a>
<a href="https://en.wikipedia.org/wiki/Oppau_explosion">obviously</a>
<a href="https://en.wikipedia.org/wiki/RAF_Fauld_explosion">store</a>
<a href="https://en.wikipedia.org/wiki/Texas_City_disaster">our</a>
<a href="https://en.wikipedia.org/wiki/2015_Tianjin_explosions">explosives</a>
<a href="https://en.wikipedia.org/wiki/2020_Beirut_explosion">safely</a>."<br>
"<a href="https://www.debian.org/security/2008/dsa-1571">We will</a>
<a href="https://en.wikipedia.org/wiki/Heartbleed">obviously</a>
<a href="https://en.wikipedia.org/wiki/Shellshock_(software_bug)">make</a>
<a href="https://en.wikipedia.org/wiki/Stagefright_(bug)">our</a>
<a href="https://en.wikipedia.org/wiki/EternalBlue">software</a>
<a href="https://en.wikipedia.org/wiki/KRACK">secure</a>."<br>
"<a href="https://en.wikipedia.org/wiki/Intel_Management_Engine#Security_vulnerabilities">We will</a>
<a href="https://en.wikipedia.org/wiki/Row_hammer">obviously</a>
<a href="https://en.wikipedia.org/wiki/Spectre_(security_vulnerability)">make</a>
<a href="https://en.wikipedia.org/wiki/Thunderspy">our</a>
<a href="https://en.wikipedia.org/wiki/Retbleed">hardware</a>
<a href="https://arxiv.org/abs/2305.10791">secure</a>."<br>
<br>
> We will teach AI to be nice to humans.<br>
<br>
Not only researchers from top AI labs have no idea how to guarantee niceness, they also have no idea how to do that at all! If they understood how to control their creations in any more precise way than "poke it with a stick and see what happens" we would not see supposedly "hidden" LLM prompts leaking day one. We would not see whack-a-mole games where internet users discover how to circumvent censorship and make AI say verboten things then AI owners patch that out repeated twenty times in a month.<br>
<br>
> But what if we train AI really hard?<br>
<br>
We already have an example of what happens then. Look at us vs. the evolution. What happens when you train a system really hard is that it learns to like stuff that is correlated with success metrics in its training environment. Then you deploy it in a different setting and it instantly goes off the rails (as judged by the trainer) because it doesn't give a shit about your stupid success metrics, it is only interested in what it likes.<br>
If you could not predict (in advance, of course, not after seeing the fact) that monkeys trained solely to maximize the number of copies of their genes in the following generations would learn to enjoy music, dancing, puzzle-solving and storytelling, and <a href="https://en.wikipedia.org/wiki/Income_and_fertility">the richer they would become the less they would be inclined to reproduce</a>, you are not qualified to predict how your AI will behave (unless you have a detailed explanatory model, which, right now, you don't).<br>
<br>
> The AI might keep us as pets.<br>
<br>
"...and provide us with everything we want and need" is an implied continuation. Even humans don't go that far. When it is convenient to us we neuter our cats and dogs. Would you be OK with, just because unmodified humans are too much of a hassle to deal with, AI cutting off, say, your sense of curiosity, instead of or in addition to physically neutering you? Also, should I mention other possibilities you ignored?<br>
"The AI might keep us as pets... because it needs guinea pigs for experiments."<br>
"The AI might keep us as pets... and torture us for fun."<br>
Now, I don't think any of these options will realistically happen, "pets" is a human concept, not a universal one; at most the superintelligence will read off everyone's mind structure/contents and put it in cold storage just in case it needs simulate something later, but still, does that look like a win to you?<br>
<br>
> Luddite! Technophobe! Hater of progress!<br>
<br>
On the contrary, if I were given a choice between being born anywhere on Earth before something like middle of the 20th century or not being born at all I'd take the second option right away. Compared to the entirety of history of humanity life in the first, second and occasionally third world today is something they couldn't even begin to imagine. Getting so far and losing everything just because we were impatient would be a monumental stupidity on our part.<br>
<br>
> Ok, then, suppose I believe you. How soon will this AGI happen? How likely will this end in disaster?<br>
<br>
The straightforward answer is "I don't know". No one does. There are timeline estimates, for example <a href="https://drive.google.com/drive/folders/15ArhEPZSTYU8f012bs6ehPS6-xmhtBPP">see here</a> for an attempt at modelling, <a href="https://www.metaculus.com/questions/5121/date-of-artificial-general-intelligence/">here</a> for an attempt at market consensus, <a href="https://ourworldindata.org/ai-timelines">and here</a> for some charts and an overview. There are probability opinions, which are all over the place.
If you ask me for my gut feeling out of pure curiosity, then today it'd be something like 5 to 20 years in the baseline scenario with no or minor interventions and overall p~0.5.<br>
<br>
> 0.5% ?<br>
<br>
No, a coinflip.<br>
<br>
> Why are you telling me this?<br>
<br>
Because I consider it both true and important. Writing a blog post is cheap and I think it will more likely help than hurt, even if the effect is tiny in both directions.
Why am I telling this to you <i>now</i> is because of two reasons. One is that I got properly scared, not through theoretical arguments, but by seeing concrete progress, first in Go, then in text processing and finally in image generation in the second half of 2022. Another is that I saw some good news recently: the AI pause letter (the first one, <a href="https://futureoflife.org/open-letter/pause-giant-ai-experiments/">from March</a>, although there was another one <a href="https://www.safe.ai/statement-on-ai-risk#open-letter">in May</a>). It was insufficient, it might have been focused on not exactly the right things and it will not get implemented, but I did not expect it until later and I did not expect so many voices of support. It meant a critical mass of realization is starting to form, which makes speaking out more valuable.
To have a chance to solve the problem first you need to be aware that the problem exists.
One advantage we have now is the internet, which lets all kinds of ideas, both good and bad, spread much faster than before.<br>
<br>
> Wait, are you suggesting this is a "forward this to ten people you know or your civilization dies in its sleep tonight" kind of chain letter?<br>
<br>
Of course not. Form your own understanding. Know how to answer questions based on your model. Then talk to others.<br>
<br>
> What do you think needs to be done?<br>
<br>
Now this is the part where it gets tricky. One problem with conventional scientific progress is that for every technology that had the potential to bite us in the ass, we let it do so, hard and repeatedly, before we learned from our mistakes. Can't have that with AI: you get only one attempt at real working AGI and once it gets smart enough to self-improve it will resist your attempts to modify it, so all mistakes you've made will evolve on their own inside that system and you don't get a chance to fix them, ever. If you are a software engineer, have you written anything more complex than fizzbuzz and got that to work with zero bugs forever on the first try? And, if that wasn't sufficiently bad on its own, we have multiple AI labs racing against each other to put more capable products on the market ahead of the competition.<br>
<br>
This is why the first step would have to remove the time pressure by an international agreement banning development of more powerful AI systems worldwide. Not forever, just until the point the fact that newly built AIs will only have our best interests in mind can be proven by formal verification and these proofs are independently checked by the mathematical community and no flaws whatsoever are found. 50-year pause would be a good start to look around and evaluate our options. Without this pause the ones who build the final invention will be the ones who cut corners, move fast and break things the most, and the only outcome that kind of attitude I can see bringing is <span style="color: #F00; background-color: #000; font-weight: bold;"> BAD END </span> with no continues.rwxhttp://www.blogger.com/profile/11961724108899856582noreply@blogger.com31tag:blogger.com,1999:blog-8200259307795996652.post-75267281922665059792023-01-15T00:32:00.014+03:002023-07-11T21:18:12.618+03:00TLMC "timeout" version, 2023.01.15If you prefer your touhou albums somewhat disorganized, but later, rather than organized, but never, then this version is for you.<br>
<hr>
<a href="https://files.catbox.moe/amxubt.torrent">Single torrent</a> : <b>1.55 TiB</b> or 1 701 548 530 091 bytes.<br>
Torrent file size is 18 106 928 bytes; contains 67 197 files.<br>
<a href="magnet:?xt=urn:btih:334a33bddf5dd15db4bcd5da2cbb1e0c514e86e5">Magnet link.</a><br>
<hr>
Added on 2023-04-25: Recent versions of uTorrent are broken (see user problems in comments) and will fail to open this torrent. Use other clients. I suggest <a href="https://www.qbittorrent.org/">qBittorrent</a>.<br><br>
This torrent contains <b>only new</b> content relative to <a href="https://www.tlmc.eu/2018/01/tlmc-v19.html">TLMC v.19</a>, so if you want a complete collection you need both.
<a name='more'></a>
There are several things you could do. Most obvious, clean and suggested solution would be to keep v.19 and this torrent in two separate directories, then union-mount contents of both to a single location. Unfortunately not an option if you're on Windows. Another way is just to point both to a single directory and create a giant mess, then if/when I release proper next version ask your torrent client to delete this torrent together with its files or ask me for a python script to delete all files belonging to a single torrent if your client can't cleanly do this itself.<br>
<br>
Collection status:<br>
- directory names (circle/artist/album/release date): mostly ok, but event names could be shortened <br>
- audio files: needs further deduplication, some albums need ctdb repair, some albums need conversion to 16/44 (anything higher is a sad reminder about the state of modern education), everything needs a uniform naming scheme to make automatic creation of cuesheets possible <br>
- digital cover versions: many albums still have better alternatives <br>
- cover scans: as good as it can get, I'm just relaying what I got 1:1, nothing to do here <br>
<br>
Added on 2023-07-11: you can view album quality scan summary <a href="https://www.tlmc.eu/p/accurip-results-v19-and-timeout-ver.html">here</a> or download original summary file without blogger code injections <a href="https://files.catbox.moe/4i8eh3.html">here</a>.
<br>
First and second columns from the left are the ( my track matches min - max / total track submissions min - max) in ARDB and CTDB, respectively, at the time I ran a scan of that album (which could easily be a year ago and is saved in the comment).<br>
<br>
If a square is dark green, then the audio is verified as OK and there is no need for duplicates;<br>
if it's light green, then there were not enough confirmations and you can send your rip and submit it to CTDB/ARDB;<br>
if it's dark red, then the audio has a mismatch and I welcome a replacement;<br>
if it's light red, then there is no match, but the audio was most likely normalized, so treat it as dark red;<br>
if it's grey, then there is no information on existing rip (either a rare CD or a web download) and your rip would help;<br>
if it's white, then it's a classification error of my script (treat as grey).<br>
<br>
A teal line is an album from v.19 (old torrent from 2018), a white line is an album from the timeout ver (new torrent from 2023), a dark grey line is a circle separator.rwxhttp://www.blogger.com/profile/11961724108899856582noreply@blogger.com92tag:blogger.com,1999:blog-8200259307795996652.post-42276763752647505672022-07-01T21:07:00.004+03:002022-07-28T21:39:16.223+03:00Looking for volunteer domain owners<a href="http://www.tlmc.eu/2020/03/bookmark-backup.html">Some time ago</a> I encountered a problem with this domain, but was able to resolve it with external help. However, current owner is no longer comfortable with that role.
<br>
Unless something is done about it the tlmc.eu domain will expire in early 2023.
<br><br>
So, once again I'm looking for a EU citizen (or an owner of a EU-based company) to acquire ownership of this domain and keep pointing it to the same location. You will need to disclose your real name and other info to the EURid and your registrar of choice.
<br>
If you're willing to help, please contact me (xmpp: rwx@headcounter.org, preferred) or the current domain owner (removed as no longer applicable).
<br><br>
Added on 20 Jul 2022: <b>The issue is resolved</b> thanks to Thyra.rwxhttp://www.blogger.com/profile/11961724108899856582noreply@blogger.com26tag:blogger.com,1999:blog-8200259307795996652.post-19845231331591865282022-03-01T09:56:00.000+03:002022-03-01T09:56:32.618+03:00A slight delay you probably shouldn't worry about.Summary:<br>
Next torrent version is in a state I'm not entirely content with, but I'm willing to just add some finishing touches, share it that way and apply fixes in the version after that. Unfortunately because of circumstances outside of my control I will have much MUCH more important things to do with my free time in the next several weeks (I wish). However, after that I expect to plan to resume working on it and make a release.<br>
<br>
Long version:<br>
I happen to live in a certain country whose senile mass-murderer at the top decided to attack and invade Ukraine.<br>
I spent most of the last week in shock (call me weak if you want), primarily because there was nothing I thought I could realistically do about it (call me slow if you want).<br>
Right now I'm looking for job offers in other countries, but because of a strong and quite literally one-in-a-million preference[*], there is only one country (US) which at the moment satisfies it and it's neither easy nor fast to get in there and two moves are as bad as two thirds of a fire... but I digress.<br>
<br>
If there's any western company which can offer permanent relocation + status adjustment, which needs a C/C++/Go/etc software engineer with 5+ years of job experience who also accidentally got a PhD in theoretical physics when he was young and adventurous, please don't hesistate to contact me at xmpp-jabber address rwx@headcounter.org or offer other contact channels here in comments that are more convenient for you.<br>
<br>
There is also a slim, but worryingly nonzero chance that the scum in the Ministry of Censorship will completely cut off internet connectivity (and then blame it on the Evil West). In that case, OOPS.<br>
<br>
One last thing. I don't think this warning is needed here, but just to be sure: pro-kremlin commenters will be shot on sight.<br>
<br>
<br>
[*] I'm not telling what it is, but you can have <a href="https://gelbooru.com/index.php?page=post&s=view&id=5023989">this cute picture</a> instead.rwxhttp://www.blogger.com/profile/11961724108899856582noreply@blogger.com66tag:blogger.com,1999:blog-8200259307795996652.post-19478347363394193082021-12-15T21:57:00.001+03:002021-12-15T22:00:25.640+03:00Quality checkRecently I ran a CueTools scan on an entire previous version of TLMC, as I was planning to do since I discovered I had this option. Not exactly the v.19, as it was already slightly dismantled (~25 albums replaced with better versions, renames applied), but something pretty close, say 99+%. The results were quite unexpected for me. If you're curious you can download <a href="https://files.catbox.moe/yjz5gs.7z">full check logs</a> (the checker might've also grabbed some extra files from bonuses) and <a href="https://files.catbox.moe/q5px14.7z">a summary</a> in both original text format and a table. Visual html+css table form uses some heuristics to transform summary text into color-coded quality representation, so it's not entirely precise. If in doubt refer to the detailed logs. You could also run the check yourself, although be prepared to let the program single-threadedly crunch numbers for a day or so, the bottleneck is the audio decode time, which runs at ~200x realtime on my CPU for the TTA codec. My logs slightly differ from the ones produced by the original Cuetools, because I patched it a little for my convenience, but the change only displays calculated offset in CTDB checks, which is not shown in the unmodified version.<br>
<br>
The summary of the results is as follows: about <b>20%</b> of all disc images fail verification. Here by "fail" I don't mean "disc was not found", I mean "was found to be different and unrepairable". Oh, and completely by coincidence the peak level of the supermajority of such rips is almost exactly 98%. And by another coincidence normalization to 98% is the default value in an admittedly turned-off-by-default setting <a href="https://wiki.hydrogenaud.io/images/3/3b/EAC_options_Normalize.png">in "Exact"AudioCopy</a>. In addition to those there are another <b>15%</b> of disc images, which also have peak level of 98% and return "disc not found" and it'd be fair to assume they would also fail to verify.<br>
<br>
Lessons to be had here:<br>
1. Not only it is your duty as a software developer to provide sensible defaults, doing only that and no more is clearly insufficient to produce reasonable outcomes. Out of 10 monkeys that see an unknown lever at least 3 will pull it and leave it there. Any option that does anything unpredictable to the dumbest possible user should be hidden behind a curtain that requires a certain intelligence level to bypass, which is enough to understand what the thing that was hidden actually does. In our particular case it could have been a domain-specific scripting language interpreter window for general purpose audio transforms, not a "please ruin my rips" on-off switch in plain sight. Of course there is another danger that by simply removing anything the unfortunate could use to shoot their feet off, instead of properly hiding it because it is easier to do so, you risk turning your software into iCrap.<br>
2. We don't know how badly mangled these rips are.<br>
Maybe they were improperly done and there are stutters or clicks or anything else that wasn't in the original CDs.<br>
Maybe the only problem is the normalization, so it is displeasing on an intellectual level, but would not be audible as a defect.<br>
3. In my defense those album versions were the only ones available, so the only alternative to including these rips was not to include them. I could not have done anything differently, had I known about it, but I wasn't aware of the problem and its extent. Now I am and so are you.<br>rwxhttp://www.blogger.com/profile/11961724108899856582noreply@blogger.com21tag:blogger.com,1999:blog-8200259307795996652.post-50854173612361207512021-12-09T17:23:00.001+03:002021-12-09T18:01:20.915+03:00Some bonuses while you waitI didn't upload a lot recently, so it came as a mild surprise to me that nyaa pantsu is dead and nyaa si closed registrations.<br>
I'll be posting some discographies of non-touhou circles that are nevertheless pretty awesome to make waiting for the big torrent slightly less annoying.<br>
As always extract cues and use cuesheet-aware player (foobar/deadbeef) to play individual tracks.<br><br>
2021.12.09 ArsMagnA (Ariabl'eyeS, -LostFairy-, Seraph) lossless music collection. <br>
<a href="magnet:?xt=urn:btih:d6480858d3c964825d3a1d4255cc725b752cf2c1"> Magnet </a> link,
<a href="https://files.catbox.moe/bw24zg.torrent"> torrent </a> link,
CUETools <a href="https://files.catbox.moe/mqeo9l.png"> verification results. </a><br>
rwxhttp://www.blogger.com/profile/11961724108899856582noreply@blogger.com7tag:blogger.com,1999:blog-8200259307795996652.post-77057433594603913432021-08-02T01:48:00.000+03:002021-08-02T01:48:53.836+03:00Why do we need a database-based filesystem.<p>Note 1: there is a small TLMC-based demo at the end of this post, I recommend to check it out.<br>
Note 2: these ideas are neither original nor new. I'm fairly sure there exist similar thoughts in written form from 20 years ago and at least three abandoned prototypes of this kind.</p>
<a name='more'></a>
<p>To illustrate the reasoning consider this toy example: suppose you are a food recipe collector. You started collecting various recipes and writing them down two years ago and was doing it since. You've managed to extract some original recipes from your aunt, mom and sister and so far you already have three broad categories of food: soup, tea and cake. Of course, you are keeping the recipes in digital form as files. As your library grows you need a good way to locate recipes instead of just dumping them all in one directory and combing through the entire list. What are your options?</p>
<p>1. Your disk filesystem suggests a natural way to do it. You create several directories, each corresponding to a certain mutually exclusive property of your files that interests you, for example a year you recorded the recipe. Once inside you choose another property, for example an author, then a food type and so on, creating a tree, until the end where you're left with only a small portion of files in every leaf. Uh, you immediately see not even one, but two problems, right?<br>
First, you will need to create duplicate author directories for each year and then duplicate type directories for each author. Just 3 layers with 3 different properties per layer and you're already at 9x duplication ("\prod_{i=2}^{N} {n_i}" in the general case).<br>
Second, this locks you in the order by which you can choose features which interest you. This scheme works as long as you go by year-author-type, but want to view all recipes by a particular author or all recipes of a certain food type? There is no easy way.</p>
<p>2. Abandon the idea of directories and store all files in one directory as a giant pile. However, give them names which encode all information about the metadata, such as "year=value1, author=value2, type=value3 recipename=value4.txt". Then use a search tool to choose whatever interests you. Unfortunately for Windows users the default search was semi-broken in XP and completely broken in 7 and onwards, but let's assume you use a working third-party one or some other OS where you have decent tools out of the box.<br>
This is better, but still problems remain. If you want to filter on two or more features at once you need to remember their exact order in the name schema, else your naive search will return nothing (and an expression which would work with either order will get monstrous very quick). You also need to create new files with properties in correct order, but here you can just clone an empty template file with all fields set to nulls. If there are several authors for a single recipe then you need to put those authors in order in your query too. Can use alphabetic sort, sure, but these small inconveniences do add up.</p>
<p>3. Yes, finally, a proper solution. Just use an SQL database, dammit. Have a table for recipes, authors, dates and what not. Have as many of one-to-many or many-to-many relationships as reasonable to describe whatever you need. Then with a single query you get anything you wanted.</p>
<p>All our problems won't go away that easily.</p>
<p>First, there is a cost to set this up. The solution is obvious if you're familiar with computers, but a novice will have to learn a whole new query language with all relevant concepts.<br>
Even more importantly, there is no integration with anything that expects to work with files/directories. If you have an application, a text editor, it wants to open files through a file open dialog, which reverts us back to the filesystem. Yes, you can drag and drop a file, but you can't meaningfully play previous/next track in the audio/video player. Saving a file is equally problematic, as you'll need to associate it with proper db structures later.</p>
<p>3+. FUSE to the rescue.<br>
Have an application that performs a mapping between SQL tables/rows/statements and a filesystem: a path becomes a query and query results become files.<br>
You don't need experience in developing kernel drivers to write a program that implements what you need. Userspace will cost you in performance, but it also means things are easier to pick apart and a mistake won't crash your system.</p>
A demo.
<p>I simply took the entire TLMC tree (snapshot of 11th of June... probably...) and looking just at its two upper layers pushed all available information into a database via a trivial regex. All fully automatic, of course, so entries with multiple circles are left as is (as compound pseudocircles), circle name hints, rename combos and decorating brackets are left as is, albums spanning multiple discs/codes are left as is and so on. You can download the resulting db and a FUSE module here:
<a href="https://sites.google.com/site/tlmcfiles/src.7z"> source </a>,
<a href="https://sites.google.com/site/tlmcfiles/db.7z"> database </a>,
<a href="https://sites.google.com/site/tlmcfiles/bin_win.7z"> windows binary </a>.<br>
There is source code for all platforms and a prebuilt windows binary. Nothing is particularly interesting about the source (it is in fact quite messy), I'm just sharing it because a sole binary is always suspicious/requires extra trust and making an executable that works on all linuxes is a pain. Don't forget to grab the database.<br>
To run this demo YOU WILL NEED : <a href="http://www.secfs.net/winfsp/rel/">WinFSP</a> on Windows, fuse3 packages on Linux, SQLite library for both platforms. Additionally, to build this you will need fairly recent gcc and friends (included in the latest <a href="https://www.msys2.org/">MSYS2</a> on Windows), fuse3 library + headers (included in WinFSP on Windows).<br>
How to run: mount your vfs somewhere as "mkdir /tmp/mnt && a.out /tmp/mnt" on linux or "a z:" on windows.<br>
How does it work: when started the program examines the databases directory. For each database [with a fixed name] it looks at its table and column names and presents them as directories. Right now it assumes certain db structure (starlike acyclic schema), in general case it could be supplied some hints by the db author. When, by entering a directory, you choose a table.column pair the program searches the db for all distinct values of this combination and lists them as the next layer of directories. You can choose a value then, this creates a constraint on results. Next you can choose another table.column, see what options you have there and so on. Once there is enough information to uniquely identify an item the full implementation is supposed to look up its corresponding file(s) and provide them as a passthrough fs (mine just returns a dummy).<br>
This is all good, however you could notice two annoyances:<br>
- Picking properties one by one is slow, you might want to navigate the directory tree according to a preset combination of properties in the order that you remember without explicitly naming every single one along the way.<br>
- Picking properties one by one is slow, you might want to see what's available in concise format where multiple properties are combined into a single choice.<br>
Both of these are easy to solve. TLMC database has two extra invisible tables. A custom field is what TLMC always used for its album directory names : the "yyyy.mm.dd [code] albumname [event]" pattern. And a preset path is a path which uses circle name as its first component and a custom field for the second. Of course you can also define your own to create any view you'd want.<br>
By the way, there are no files, this is just a demonstration of directory structure a database-based filesystem could have. Also, the whole thing is a quick hack job, so it's relatively fragile and could break if you try to hand-feed it deliberately crafted confusing inputs. It is also criminally slow to list some virtual directories in comparison to regular filesystems, but as long as it's faster than human reaction time you can afford to pay that price.</p>
<p>Q: So, after all this we can do the same thing we always could, just in a more complicated way?<br>
A: Not exactly. We can do the same thing and much more. For example you can trivially add transliteration for circle names (or album names or event names, but I skipped that). Well, more like translation. Well, even more like whatever google thinks you think you wanted, but you get the idea, this was also automatic. Then instead of seeing a list of squiggly blobs [in case you can't read japanese] you'll see familiar english letters. Or you can filter on any property and browse in any order you'd like:<br>
- Pick "event.name" first, "circle.name" second and see which circles participated in the event you chose.<br>
- Pick "album.year" first, "event.name" second and see which events happened in the year you chose. Strictly speaking this abuses the fact that album release date almost always coincides with event date, but date fields could be added to the event table to disambiguate.<br>
- Pick "circle.name" first, "album.year" second, "custom field" third and see all albums by a circle released in a certain year.<br>
Okay, there's not a whole lot of combinations now, but it does add some convenient options. Pick "prepared path 1" to view albums in historical-TLMC-like way.
</p>rwxhttp://www.blogger.com/profile/11961724108899856582noreply@blogger.com6tag:blogger.com,1999:blog-8200259307795996652.post-21344168779454358802021-06-28T14:15:00.006+03:002021-06-28T18:15:52.046+03:00There are no good browsers left.<p>I actually intended to make this post a year or two ago, but then kept putting it off. Now I stopped doing that, so here you are.<br>
What is this about? I don't mean to criticize how bloated the browsers have became, even if it's true. I don't mean to write about how Firefox always shuffles UI elements around or how Chrome spies on its users. Just the very simple, most basic thing. <b>Fonts</b> and font rendering.<br>
Make sure to set your browser page zoom to 100% to avoid browser scaling image distortions.</p>
<p>Here's a <a href="https://sites.google.com/site/tlmcfiles/font_test.html">tiny piece of html</a> for this test.</p>
<a name='more'></a>
<p> 1. This is how the text used to look for every Firefox user (on Windows) and still looks on my end. First image is 1x and the second is 8x nearest-neighbour magnification so you can easily see the details. Right-click and open in a new tab or set page zoom to 100% or download the images to view them in original size.<br>
<img src="https://sites.google.com/site/tlmcfiles/1x_ff_cairo.png" style="border: 1px solid red; float:left;" alt="FF Cairo 1x">
<img src="https://sites.google.com/site/tlmcfiles/8x_ff_cairo.png" style="border: 1px solid red; " alt="FF Cairo 8x NN"><br>
Notice how every single entirely horizontal and vertical line is perfectly aligned with the pixel grid. Antialiasing for the round parts is there, it's unavoidable, but it doesn't turn on until some threshold font size. Smaller fonts use exact hinting and stay at 100% contrast, larger fonts smooth only the curved segments. This is Firefox until version 57 with forced Cairo font backend. This is how it did and should work.</p>
<hr>
<p> 2. With some bullshit justification typical for the Firefox team they completely removed good rendering option. This is how the text looks now for any user of a more recent Firefox (ok, I made the screenshots years ago on version ~60-80, but it could have only become worse at this point - I'm not reinstalling some new version and the test html is there for you to try if you think I'm wrong).<br>
<img src="https://sites.google.com/site/tlmcfiles/1x_ff_skia.png" style="border: 1px solid red; float:left;" alt="FF Skia 1x">
<img src="https://sites.google.com/site/tlmcfiles/8x_ff_skia.png" style="border: 1px solid red;" alt="FF Skia 8x NN"><br>
Small font is still the same, but larger fonts got worse. Maybe the distances between letters are now correct floating point values that were prescribed by the dumb new engine, but it cost us vertical lines which became gray and/or blurry. To say directly and honestly: this rendering sucks. The moment I saw it was the moment I stopped updating FF and reverted to the last known good version.</p>
<hr>
<p> 3. What about the browser with the largest market share, Chrome? Ok, ungoogled chromium to be precise, but in this regard it's the same thing. Well...<br>
<img src="https://sites.google.com/site/tlmcfiles/1x_chromium.png" style="border: 1px solid red; float:left;" alt="Chromium 1x">
<img src="https://sites.google.com/site/tlmcfiles/8x_chromium.png" style="border: 1px solid red;" alt="Chromium 8x NN"><br>
Like, seriously?? This is the best that google(!) could come up with? It isn't just shit smeared across the screen: the blurriness of the whole thing makes you want to claw at your face and vomit your brain through the eyesockets. How could anyone look at this hideous abomination for more than 5 seconds and think it's even remotely acceptable not to immediately wipe the software that decided to draw this from your machine?</p>
<hr>
<p>Potential objections:<br>
> "Just buy a high-dpi monitor, bro"<br>
This is doubly retarded for the following reasons. First, it requires me to fork over some cash for the hardware to solve the problem that was not merely preventable in software, but rather was created in software in the first place. And second, even if your new resolution is double the old in each dimension for the same physical size, then that would only halve the width of the blurred gray edges, not eliminate them entirely. So, instead of solving the problem this provides a half-assed band-aid solution that costs money and doesn't even work. **** you very much, but I'd rather keep my old display and my old browser that work perfectly fine together.</p>
rwxhttp://www.blogger.com/profile/11961724108899856582noreply@blogger.com16tag:blogger.com,1999:blog-8200259307795996652.post-10541979479971224592021-06-11T20:58:00.005+03:002023-05-23T23:24:12.966+03:00TLMC v.20 PRE-RELEASE post<p>
The bad news: v.20 is still several months away.<br>
The good news: v.20 is only several months away.<br>
This is NOT YET a release, but we are fairly close. Here's how you can help:</p><a name='more'></a>
<p>I will be posting a list of albums I've managed to collect from various sources so far with updates from time to time until the torrent release.<br>
- If there is something in the list that should not be there (unrelated albums), say so.<br>
- If you have something that is not in the list (and not in the latest tlmc, of course), do share. Make sure to check these things first:<br>
--- Ctrl-F my latest list and the comment section of this post to check if your album is not a duplicate. Search for smallest unique substrings, in case spelling is slightly different or wrong.<br>
--- Good scans are always in short supply. Even if the album is in the list it might be missing scans. If you have the files, err on the side of asking/sharing.<br>
--- I'm monitoring doujinstyle, so no need to announce or offer copies of their new uploads (unless yours is better, for example +scans or +log or 16/44). However, I could miss something out of their ~16k links, especially if it was not marked as touhou, so a double-check would be appreciated.<br>
- If you can download from baidu, then check if acgjc.com touhou subsection has anything new.<br>
- If you can download from baidu, please reupload links that I or anyone else will occasionally post here.<br>
- If you cannot download from baidu, but have some baidu links of new material, then post and hope someone who can will reupload them.<br>
The "new album" list could not only grow, but also shrink, if I discover files that fail decoding, have wrong content, are unrelated, etc. I don't expect many such problems, though.</p>
<p>In <del>several weeks</del> due time I'll add a list of all files and an archive of cuesheets. I'm still adding digital covers for the new albums and would like to go through all of the old albums without scans as well, if time permits.<br>
I stripped personal comments temporarily appended to directory names, so some entries in the "new album" list may appear as duplicates of v.19, no need to point that out. This is not a problem: these are either new scans, additional scans, alternate scans, extra logs, or simply dupes I didn't compare yet.</p>
<p>If you're aware of a production-quality program that can extract JPEG content from TIFF files without bitmap decode-encode steps, let me know.</p>
<p>Every album that is assembled from separate tracks will have a "REM COMMENT built from tracks" or similar comment in its cuesheet. This is not bad and doesn't mean much on its own, it's mostly for bookkeeping purposes. Every single album built from tracks that had a single-file duplicate (with maybe one or two, literally, exceptions) was either content-wise bit-identical to its dupe or merely shifted by difference in drive offsets.</p>
<p>The smallest addressable unit in cuesheets is a frame, which is 1/75-th of a second or 588 samples (2352 bytes). If any length of a track in the album is not an integer amount of frames, then every track in that album cuesheet gets a sample-precision start and end comments, which makes it possible to recover exact audio data content of the files used, should you ever want it. It also means that either (far more likely) files were mastered directly for digital distribution, bypassing CD creation step, or (far less likely) some shitty software was used in the creation of that CD rip.</p>
<p>About 5% of files are problematic, with either sampling depth of more than 16 bits or sampling frequency of more than 44.1-48 kHz. These are mostly bandcamp releases. I'm still undecided on how to proceed with them. Will add a list a bit later.</p>
<p>If you were ever under impression that flac or lossless in general means untouched CD audio rip, sans an occasional artifact or two, then I'm here to disappoint you. At least one album that almost made it into v.20, but got accidentally caught by me, was postprocessed by the ripper. For more details you may read a comment that alerted me of this fact <a href="http://135.181.29.38/comiket/comiket.txt">here</a>, search for "destroyed". I downloaded both versions, the difference is clearly audible. Thanks to the "people" who decide to do this as if the audio weren't already loud to the point of clipping often enough, we can't be sure albums are really correct CD rips if they don't contain rip logs. Old Share/PD rips are less suspect because the average intelligence on the internet never goes up. I don't think it is a frequent occurrence, but still, if you reshare something then DON'T DISCARD THE RIP LOGS. I might try to scan all albums with cuetools if I find its batch mode, although rare CDs would naturally have no db submissions to compare against.</p>
<p>I still don't have a clear policy on sub-titles, which are album names transliterated into english (frequently in uppercase) and appended to the main title. For now I follow local consistency rules.</p>
<p>Many event labels are missing (trailing "[]") or have inconsistent spelling, this also creates perceived dupes. Ignore, it will go away soon.</p>
<p>I have an idea about a secondary supplementary torrent, which would include non-touhou albums of circles if they make up significantly less than half of that circle repertoire, so that by combining both you would get complete discography of a circle. It is definitely not happening any time soon. However, I don't delete unrelated albums I download (or find in and remove from the torrent), just safely put them away, so keep that in mind.</p>
<hr>
<a href="https://sites.google.com/site/tlmcfiles/dirs_v19plus_level2_2021-06-11.txt">old </a> v.19+ list for convenience (~6.5k lines), updated on 2021-06-11<br>
<a href="https://files.catbox.moe/eezuqk.txt">
v.19+ list for convenience (~6.5k lines)
</a> last updated on 2021-09-14
<br>(includes proposed moves, renames, deletions)<br><br>
<a href="https://sites.google.com/site/tlmcfiles/dirs_v20delta_level2_2021-06-11.txt">old </a> v.20 tentative additions (~3.3k lines), updated on 2021-06-11<br>
<a href="https://files.catbox.moe/6jx936.txt">
v.20 tentative additions (~3.5k lines)
</a> last updated on 2021-09-14<br><br>
<a href="https://sites.google.com/site/tlmcfiles/ren20beta_2021-06-11.py">old </a> v.20 proposed moves and renames, updated on 2021-06-11<br>
<a href="https://files.catbox.moe/7czgsk.py">
v.20 proposed moves and renames
</a> last updated on 2021-09-14<br><br>
<a href="https://files.catbox.moe/nojfdr.diff">
v.20 diff vs v.19 (10k lines)
</a> last updated on 2021-09-14<br><br>
rwxhttp://www.blogger.com/profile/11961724108899856582noreply@blogger.com192tag:blogger.com,1999:blog-8200259307795996652.post-59081103475543309032021-04-21T18:43:00.003+03:002021-04-21T18:54:39.083+03:00Booth.pm corrupts your downloads.If you ever bought direct downloads from them make sure to check that your files are error-free, especially if your downloads were slow.<br><br>
Today I noticed that half of the files in four free albums by "クロネコラウンジ" did not decode properly. I redownloaded them all and one of the new files was still broken, but another attempt got me a good version. It was the only file that took about two minutes to get (today's first attempt), compared to several seconds for the rest, which suggests that under heavy load the file-serving part of their webmonkey code hiccups and starts sending you wrong file parts. Looking at the broken files in hexeditor you would notice content starts repeating in some random place (the starting position of the first part that gets repeated is aligned at 1 MiB boundary).rwxhttp://www.blogger.com/profile/11961724108899856582noreply@blogger.com3tag:blogger.com,1999:blog-8200259307795996652.post-46427282384145444732021-01-10T12:17:00.004+03:002021-02-23T17:47:09.665+03:00Princess Connect<p>
Important notice 1: TLMC pre-release post soon(tm), the torrent several weeks after that post. Don't ask when, it's being worked on.
<br>
Important notice 2: Princess Connect Re:Dive [global] is available both in "32-bit" (ARMv7) and "64-bit" (ARM64) versions, don't listen to anyone claiming otherwise.
Just visit some direct-from-google downloader <a href="https://apk.support/apk-downloader">like this</a>, and pick SG Grand Prime (for 32) or SG Note 9 (for 64).
<br>
Important notice 3: Leapdroid won't run PCRD (some OpenGL surface issues). Bluestacks emulator, which I saw being recommended by many, contains a metric shitton of malware out of the box, be advised.
<br>
</p>
<p>
Just a short observation - if you feel like peeking at exact values the game uses internally, you should view its db (which is located in "files/manifest" app settings subdirectory; the one with the sqlite header).
If, for any reason, you just want to refer to some data there without exporting it yourself, you may use <a href="https://github.com/gf-db/gf-db.github.io/tree/master/pcrd/db_dump">this dump</a>.
I'm not sure I'll be sticking with the game for a long time, so I can't promise regular updates yet.
<br>
</p>
<p>
Based on the current db the game will be officially released on
<a href="https://github.com/gf-db/gf-db.github.io/blob/7f14793bd73672d451416e9720f97cfd09190405/pcrd/db_dump/login_bonus_data.box">19 Jan 2021</a>
and clan battles will start on
<a href="https://github.com/gf-db/gf-db.github.io/blob/7f14793bd73672d451416e9720f97cfd09190405/pcrd/db_dump/clan_battle_period.box">10 Feb 2021</a>.
<br>
First "focused gacha"' will be
<a href="https://github.com/gf-db/gf-db.github.io/blob/7f14793bd73672d451416e9720f97cfd09190405/pcrd/db_dump/gacha_data.box">rateup for Djeeta</a>.
</p>
<p>
20 Feb 2021 edit: Stopped updating the db dump on github.<br>
The game turned out to be pay-to-win trash, wouldn't recommend it to anyone.
</p>rwxhttp://www.blogger.com/profile/11961724108899856582noreply@blogger.com6tag:blogger.com,1999:blog-8200259307795996652.post-41471979602829617542020-05-08T14:04:00.000+03:002020-05-08T14:12:28.276+03:00Proper GF battery chart<p>
Yet another thing that was slightly bothering me about GF for a long time was wrong information about the effect dorms and their comfort have on collected batteries that is repeated everywhere.
It was bothering me only slightly because I never had enough gems to go past 3 dorms. However, some time ago I got rich fast (in gems, not real life), so I decided to fix the issue.
<p>
<a name='more'></a>
<img src="https://sites.google.com/site/tlmcfiles/chart_dots.png" width="100%"/>
<p>
Here is the chart. Horizontal axis is the sum of all dorms' comfort in thousands, vertical axis is batteries gained in 24 hours. Line color ~ dorm count.
<br />
<a href="https://sites.google.com/site/tlmcfiles/battery_stats.tsv">Here is the raw data</a>, if you're curious. Some extrapolations were made.
</p>
<p>
Note: if you enter any base facility between collections during the time condenser is active, then the gather time will be split in two and the amount you get will possibly be one less because of rounding down.
</p>rwxhttp://www.blogger.com/profile/11961724108899856582noreply@blogger.com0tag:blogger.com,1999:blog-8200259307795996652.post-46832424485537176542020-03-18T00:57:00.003+03:002020-03-18T00:57:41.892+03:00Bookmark a backupYesterday I got an email that looks like a fishing attempt. Bla-bla, validate your identity or else we'll suspend your domain, bla-bla. Just in case it is not a scam I recommend you to save a backup name that leads to this blog: <a href="http://nameless--fairy.blogspot.com">http://nameless--fairy.blogspot.com</a> (note the double dash).rwxhttp://www.blogger.com/profile/11961724108899856582noreply@blogger.com12tag:blogger.com,1999:blog-8200259307795996652.post-46569242482753024402019-11-06T11:52:00.000+03:002019-11-09T11:58:36.613+03:00Significantly reduced GFDB precisionStarting from 03 Nov 2019 the frequency of the log data updates returned by the game server changed once again: now refresh interval is more than once per hour (~20 updates in 24 hours). If this is a permanent change then it's rather bad news for anyone expecting meaningful statistics from EN GFDB in the future.
<br><br>
UPDATE: On 06 Nov everything got back to normal.rwxhttp://www.blogger.com/profile/11961724108899856582noreply@blogger.com0tag:blogger.com,1999:blog-8200259307795996652.post-38266837786785625382019-08-05T00:27:00.000+03:002019-08-05T00:36:09.847+03:00On modern software<p>
Today I noticed that <a href="https://gf-db.github.io">gfdb</a> updates were broken again. It was the third time it happened, so I knew this deserved a post.
</p>
<p>
I caught first update breakage purely by accident. One day I looked at the process list and noticed a chain of git commands sitting there doing nothing. Apparently at some point git decided it was time to do garbage collection, so it ran "git gc --auto", which, for reasons yet unknown to me, could not complete in 50 minutes on a fairly small and completely linear repository. This step should have happened before the push took place, but since it just hung there (with no cpu/disk activity, by the way) it was killed by task scheduler every hour to start new task, which repeated everything.
This was going on for 5 days before I noticed. I run "git gc" manually and it finished in about a minute, cleared the clog and all was good (for a while).
</p>
<a name='more'></a>
<p>
Second time was exactly the same as first, except I was already aware it could happen, so I reacted sooner. Internet search brought nothing: either it was a new bug, or it was specific to some part of my configuration. Since I (mistakenly) already dismissed it I had to wait for another chance to debug and see what was happening.
</p>
<p>
However, today's problem was different. Call stack was not the same and had bash prompt for input at the top instead of gc command. After I killed it and manually tried to push all accumulated updates git asked me... for a username? What? Did github ban me, so my user-pass no longer worked? But why?
I tried to login into github. <i>Blah-blah, we see your device for the first time, how about you go fetch verification code we sent to your email?</i>
Oh great, fuck you very much, another service turned """""user-friendly""""". But whatever, here's your code.
</p>
<p>
Hmm, no notifications, the repo is there, everything looks pretty normal.
I tried pushing again from the console. Typed the username and copypasted the password.
</p>
<p><code>
>remote: Invalid username or password. <br>
>fatal: Authentication failed
</code></p>
<p>
What? Did I just mistype 5-letter username or miscopy a doubleclicked string? Try again:
</p>
<p><code>
>Enumerating objects: 14190, done.
<br> >Counting objects: 100% (14190/14190), done.
<br> >Delta compression using up to 12 threads
<br> >Compressing objects: 100% (1834/1834), done.
<br> >error: RPC failed; HTTP 401 curl 22 The requested URL returned error: 401
<br> >fWatal: the remote end hung up unexpectedly
</code></p>
<p>
What?? Looks like github is having problems, should I try again?
<p>
<p><code>
>remote: Invalid username or password. <br>
>fatal: Authentication failed
</code></p>
<p>
And then after one more attempt it accepted the changes I tried to push.
</p>
<p>
But what about git? Where is the password that was saved what, 3 months ago, and reused every hour since then? I went searching and encountered <a href="https://stackoverflow.com/questions/35942754/how-to-save-username-and-password-in-git">this magnificent piece of information</a>:<br>
>"git pull" will fail, because the password is incorrect, git then removes the offending user+password from the ~/.git-credentials file
</p>
<p>
Let me repeat that, so you can enjoy it one more time: whenever git discovers that any stored user+pass combination doesn't work it simply erases it from saved credentials file.
I checked the file - yeah, 0 bytes. All puzzle pieces are in place.
</p>
<p>
What did we learn today?
</p>
<ul>
<li>Shame on you, git. I did not expect this systemd-level bullshit from you, of all programs.</li>
<li>Shame on you, github. Having transient authentication problems (caused by service overload, network partitioning or something else) is understandable, returning wrong error statuses and not notifying affected users is not.</li>
<li>Modern software (and software as service doubly, if not triply so) ALWAYS needs to be propped up by humans. Even if you setup everything correctly and leave it unattended, then sooner or later, mostly sooner, something will fail.</li>
</ul>rwxhttp://www.blogger.com/profile/11961724108899856582noreply@blogger.com7tag:blogger.com,1999:blog-8200259307795996652.post-30741911928012271572019-07-23T21:20:00.000+03:002019-08-06T11:31:33.226+03:00First Girls' Frontline guestopocalypse<p>
Today during the maintenance I noticed an error in craft scraper log that I've never seen before: all login tokens were being explicitly rejected. This slightly scared the hell out of me, it could mean that my account, which I kept as a guest, was also in danger. After the mainenance concluded the error message didn't change, my own account was also locked out.<br>
Well, shit.
</p>
<a name='more'></a>
<p>
I found Sunborn support address and sent them an email. At the same time I started to wonder how many melons and monthly rewards I'll miss before I get my account back, if I get it back, that is. However, first support reply arrived in <b>6 minutes</b>. They requested information about my account that only the owner could know. Completely accidentally I had a table with all my doll/fairy levels and skill levels, so I sent them a screenshot of that. Then they asked for email to bind, sent a link there, the link gave me the password, I put all that into GF client and it logged in. The whole exchange took less than an hour.
</p>
<p>
Results:<br>
Sunborn tech support: <font color="gold"><b>S-</b></font>. They get "S" for speed/success in helping and "-" for confusing me by replying to my response to their email request with the same email (was my request erroneously routed to two supoort guys at once?) and then sending me two recovery URLs.<br>
Sunborn backend operations: <font color="teal"><b>C</b></font>. At least they didn't wipe all guest accounts completely. Unfortunately this level of total incompetence is so widespread in the industry, that the company didn't even find it necessary to apologize for their mistake.<br>
</p>
<p>
I also had to reroll scraper accounts, which cost a couple hours of log downtime, but luckily I got all of them in just 15 attempts, with 30 being the expected value.
</p>rwxhttp://www.blogger.com/profile/11961724108899856582noreply@blogger.com2tag:blogger.com,1999:blog-8200259307795996652.post-83433680421558581582019-04-21T14:51:00.000+03:002019-08-04T23:43:16.490+03:00Girls' Frontline production anomaly.<p>
Today I was looking at the GFDB and noticed something strange: there were some results that just weren't supposed to happen:<br>
1 (one) <a href="https://gf-db.github.io/gfdb/gfdb.html?type=tdoll&id=213&epoch.tdoll=19.0&sort.tdoll=[{%22sort_column%22:%22mean%20%%22,%22dir%22:-1}]"> [SMG] C-MS </a> each from HG recipe 130-130-130-130 and AR recipe 97-404-404-97,<br>
1 (one) <a href="https://gf-db.github.io/gfdb/gfdb.html?type=tdoll&id=228&epoch.tdoll=19.0&sort.tdoll=[{%22sort_column%22:%22mean%20%%22,%22dir%22:-1}]"> [SMG] 100 Shiki </a> each from HG recipe 130-130-130-130 and AR recipe 91-400-400-30,<br>
1 (one) <a href="https://gf-db.github.io/gfdb/gfdb.html?type=tdoll&id=215&epoch.tdoll=19.0&sort.tdoll=[{%22sort_column%22:%22mean%20%%22,%22dir%22:-1}]"> [AR] MDR </a> from SMG recipe 400-400-91-30.<br>
</p>
<a name='more'></a>
<p>
I could think of several reasons for that:
<ul>
<li> Magic </li>
<li> MICA's shady new crafting formula </li>
<li> Tester-kun is fooling around again </li>
<li> It's an Arknights conspiracy </li>
</ul>
</p>
<p>
As you probably know, templates like that are forbidden. At first I was afraid that my SQL import process failed because I tried to speed it up so much that what was left of a full json parser was a basic tokenizer which didn't even check field names, assuming that their order was deterministic and could only change because of some server maintenance or update. Fortunately I keep complete raw logs. One search later I confirmed that no, import was doing fine, and the anomaly shows up there too. If you're curious, here are the crafts in question:
</p>
<p>
<code>
{"id":"11027121","dev_type":"0","user_id":"52674","build_slot":"1","dev_uname":"Lani","dev_lv":"63","gun_id":"228","mp":"91","ammo":"400","mre":"400","part":"30","input_level":"0","item1_num":"1","core":"0","dev_time":"1555766798"}
{"id":"11002254","dev_type":"0","user_id":"91144","build_slot":"1","dev_uname":"AdultNeptune","dev_lv":"126","gun_id":"215","mp":"400","ammo":"400","mre":"91","part":"30","input_level":"0","item1_num":"1","core":"0","dev_time":"1555755557"}
{"id":"10986940","dev_type":"0","user_id":"599614","build_slot":"1","dev_uname":"uwux3usowarm","dev_lv":"33","gun_id":"213","mp":"130","ammo":"130","mre":"130","part":"130","input_level":"0","item1_num":"1","core":"0","dev_time":"1555750872"}
{"id":"10554813","dev_type":"0","user_id":"5795","build_slot":"1","dev_uname":"Arveene","dev_lv":"117","gun_id":"213","mp":"97","ammo":"404","mre":"404","part":"97","input_level":"0","item1_num":"1","core":"0","dev_time":"1555806667"}
{"id":"10468473","dev_type":"0","user_id":"317635","build_slot":"1","dev_uname":"Eoneo","dev_lv":"158","gun_id":"228","mp":"130","ammo":"130","mre":"130","part":"130","input_level":"0","item1_num":"1","core":"0","dev_time":"1555751142"}
</code>
</p>
<p>
There is nothing particularly suspicious in the data: all userids are different (they do belong to 2 out of 10 shards, but that's not quite enough), times are quite far apart (doesn't look like particle shower bitflips and AWS hardware should have ECC anyways), no magic numbers indicative of overflow of some kind. I don't know what caused this yet.
</p>rwxhttp://www.blogger.com/profile/11961724108899856582noreply@blogger.com0tag:blogger.com,1999:blog-8200259307795996652.post-48261469145295576962019-04-13T20:09:00.000+03:002019-08-04T23:43:26.642+03:00Girls' Frontline statistics, parts of the missing chapter.<p>
Alright, you primitive screwheads, listen up. See this? <a href="https://gf-db.github.io/">gf-db.github.io</a> This is your new GFDB.
</p>
<a name='more'></a>
<p>
Well, to be fair, it's somewhat unfinished, but better than nothing, right?
</p>
<p>
There is one more thing that complicates straightforward analysis of rates during various periods that I didn't mention in the previous post. "dev_time" parameter is the time you take the item (doll, equip or fairy) out of production. However, since it is known in advance what will be produced, because most items are 1:1 mapped to their production times, probability roll happens when you start the construction. Thus some rolls will leak: they will be produced using rates of one crafting period, but will be taken out and appear in some subsequent period. You can exclude such crafts at the cost of slightly reducing overall amount of samples: for every user for every their build slot discard first craft in every production period. The effect we're talking about here is not that big: if you run dumb query that doesn't account for it, you introduce bias in estimated rates proportional to the difference between true period rates divided by the average amount of crafts per user during second period.
</p>
<p>
Yet another effect to look out for is regression towards the mean. Suppose you took the list of recipes for any particular item and arranged it in descending order by mean value of item probability estimate, then noted top ones. If you came back later after more data was gathered more likely than not you would notice that all supposedly "good" recipes' mean values fell. This does not mean that fresh new recipes get "originality boost" or something like that. Since by their very nature results are random in any distribution you will get recipes that perform both better <i>and worse</i> than their true rate on a small sample size and with more data all these fluctuations will smooth out.
</p>
<p>
A sidenote: there are times when one could begin to wonder if all the hate of microsoft products is really warranted and not a mindless parroting of another camp's fanboys. And then usually something like this happens:
I needed to add regular statistic updates for the site. Should be easy, right? Since I run the capture on my main machine, and it has to run win7 because games... open Task Scheduler, create a task, done? Well, I did that and for some reason, call it intuition or curiosity, added timestamp logging, to see how much time updates take. When launched manually 1 hour of files is typically added in 2 seconds and the process to recreate statistics takes about 20 seconds, which is a lot and is a result of unoptimized queries/indices doing full scan on a gigabyte-sized database and the fact that I wrote it once and then used to run it every other month, but that can be easily fixed and is not the point now. So, testing showed that the data was added, generated, so I left it alone. And then after about half a day I peeked at the log and did "O_O" face when I saw update times of 20-40 minutes (?!?!?!1).
After a quick... ducking? (like googling, but without the privacy invasion part) I found out that (in increasing order of retardation): <ul>
<li> all processes launched from windows task scheduler run with lower CPU, I/O and memory priority </li>
<li> there is no setting in the GUI to change it </li>
<li> there is a way to change it that involves exporting the job to XML, editing "priority" variable, and importing it back, but </li>
<li> this variable affects both CPU and I/O priority together and there is no way to set memory priority at all </li>
</ul>
And even if we adjust CPU+IO priority to that of an interactively-launched process and leave memory alone, because it should only affect how soon our process' data gets evicted from the page cache under memory pressure, the updates still take 40-60 seconds. Why does that happen on an idle machine is still a mystery to me. So yeah.
</p>
<p>
Suppose you have a nearly complete log of crafts with all the data I described in the previous post. What interesting/useful information can you gain from it? Let's see: <ul>
<li> A long list of <a href="https://sites.google.com/site/tlmcfiles/animeti.png">real user-supplied names</a>; more than 175k entries at the moment of this post. </li>
<li> User level as a function of time with 1 day or better resolution for each player (assuming they do daily crafting). Typical/fastest/slowest growth rate, average time it takes to levelup at level x for entire server population. </li>
<li> History of all name changes. For example, there are 3 users who changed their name twice and 236 users who changed their name once at the moment. </li>
<li> Ability to do username-to-id lookups for dorm-visiting or <i>other</i> purposes. </li>
<li> Item/class/total craft rates as a function of time. Popularity of various recipes. </li>
</ul>
</p>
<p>
[I was supposed to put cool-looking charts here, but then got lazy. I might revisit them if anyone is interested.]
</p>
<p>
Chinese-quality code found in game (I mean, other than UI "responsiveness"): <ul>
<li> Client-side name validation. Your name cannot have any of these strings as a substring (case-insensitive comparison): "delete", "drop", "truncate", "set", "database", "table", "field", "alter", "select", "update", "insert". I don't know whether to laugh or cry here. </li>
<li> On "Combat" page accessible from the start menu if you rapidly switch between any 2 selector buttons (Combat mission, Logistic support, Combat simulation) game will reliably crash. </li>
<li> Not joking, happened to me once: the game crashed when I opened Formation window and started rapidly removing dolls from an echelon one by one. When I relaunched it I found myself in the defense drill battle. Lost 5 extra energy to this. </li>
<li> When moving from dorm to dorm using the Next button right after you press it everything starts looking pretty crappy, especially noticable on the condenser and wall portraits. No wonder why - they take a screenshot of the scene to slide it and for some reason (nobody gave a shit, that's the reason) it is temporarily saved as jpeg with high compression, so all small details are blurred and artifacts are scattered all over. If you're playing on a phone it might be less noticable because angular pixel size is smaller for typical viewing distances. </li>
</ul>
</p>
<p>
Interesting observations: <ul>
<li> There are no "Oath" method/variable names in the game code, only "Marry" / "Wedding" ones. </li>
<li> Christmas bean bag chair furniture piece can be used as a bed, but only by G11. </li>
<li> I went and plotted 5* SG rates from the most popular recipe over time. On 2019-02-26 rates <a href="https://sites.google.com/site/tlmcfiles/sg_rates.png">effectively tripled</a>. </li>
<li> On 2019-03-05 (after the update) development log update frequency was increased from once every 10 minutes to once every 5 minutes, except last 4 hours of each server day where it was decreased to once every 40 minutes. On 2019-04-02 (after the update) they rolled it back. </li>
<li> On the chart image in the previous statistics post perceptive readers would notice a small bump during the maintenance. It is not an artifact. Looking at the database these are crafts by a single user named "Hiden", apparently as tests on the production server. You may visit this tester-kun at UID 4422 and say hello (not so <i>hiden</i> now, eh?). </li>
</ul>
</p>
<p>
I trust you, reader, not to be a retard, who ignores everyting I've written and rushes to craft recipes with the highest average ignoring low epoch capture ratio and/or high stdev due to low amount of crafts.<br>
However, even if you are one, then no significant harm is done, because the landscape looks pretty smooth to me, so the most impact you can have on your results is to simply do more crafts while keeping resource efficiency in mind.
</p>rwxhttp://www.blogger.com/profile/11961724108899856582noreply@blogger.com6tag:blogger.com,1999:blog-8200259307795996652.post-64304163407156466232018-11-24T01:33:00.000+03:002019-08-04T23:43:35.927+03:00Girls' Frontline StatisticsPutting a bit of "everything else" in the blog. This post is about a mobile game and applied statistics.<br />
There will be absolutely no mentions of touhou (other than this one) in the post. Estimated time to read: 15 minutes.<br />
<br />
<a name='more'></a><br />
<br />
<span style="white-space: pre;"> </span><b>Part 1, introduction.</b><br />
Traditional gacha mobile games have a simple core mechanic and a number of units that participate in it. Several units are given to the player as starting units, some more can be farmed from maps, but the majority of most desirable, stronger units, come from the gacha - an RNG-powered slot machine that eats game premium currency (real money disguised as another game parameter in a pretty successful psychological trick to mask true monetary cost of player addiction). As the player progresses further in game gacha units become effectively mandatory - only the most hardcore players can do the "no gacha" challenge. Then there is also the collection aspect...<br />
<br />
Girls' Frontline is what I'd call a second generation gacha mobile game. The trend started with Kancolle, if I had to guess. Other than that, the list includes Azur Lane, Cuisine Dimension (no english version yet) and maybe some other smaller titles. Their (our?) gacha is different - all unit and equipment crafts utilize resources that can be freely farmed in-game. Premium currency is still there, the game is not a charity after all, but it can be spent either to expand your infrastructure (free handouts can cover that fairly well), to roll cosmetics gacha with various costumes and decorations that are not necessary "to win" or to buy resources at <del>ridiculously</del> high implied time-to-money conversion rate<a href="#fn1" name="fn1ref">[1]</a>. There are still whales as gacha rates for rare cosmetics items are rather low (they wouldn't be rare otherwise) and the game seems to be doing fine profit-wise. Overall such games feel much more "fair" (for whatever subconscious definition of fairness) for both f2p and paying players, that's one of the reasons I picked it up.<br />
<br />
The game has 4 types of basic resources and a crafting system which can be fed user-defined quantities of each resource and then produce units (dolls, in game vernacular) or equipment. Results are random, but there are certain consistent patterns that players can notice if they experiment with various resource amounts and ratios a lot or just read and confirm experiment results of earlier players. Game designers/admins give no information about the effects recipes (resource combinations) have on construction results, making figuring out best recipes another interesting part of game experience. Here "best" mean ones that give highest rates either for dolls/equipment of highest rarity or specific rare dolls, because in the process of hunting for rares you will naturally produce more than enough of common ones.<br />
<br />
Typical consistent patterns mentioned earlier are hard cutoffs, which guarantee that you will exclude certain categories of stuff from crafting if you put in amounts of certain resources below or above some threshold. For more details, you may visit a calculator, <a href="https://aaeeschylus.github.io/main.html">such as this one.</a> More interesting question is what kind of effect does varying amounts of resources within these bounds have on individual item appearance rates. Theoretically game designers could have made rules extraordinarily complicated, but I suspect that realistically there are only 3 possibilities: we need to find out whether crafting probability landscape<a href="#fn2" name="fn2ref">[2]</a> is spiky, smooth or flat.<br />
<br />
Amount of crafts a single player can do with their limited resources is woefully inadequate to obtain reasonably precise estimates. Manually collecting results from several players is a logistics nightmare and a futile exersize in filtering out deliberate misinformation and accidental logging errors. If that was all we had I'd give up day one. However, fortunately for the curious types, developers left a small hole to slither through. The game features a PRODUCTION LOG, which shows recent recipes used by players and what did they obtain. Intercept game's network connection, figure out the API and results format, and then repeatedly query the server and obtain full and complete log of all crafts for you to analyze - the concept is obvious and trivial. The devil is, as always, in the details. It's a small project that can take some time to get right. It requires you do sit down and actually do stuff. Instead of that, how about some instant gratification in the game?<br />
<br />
That's how I thought, putting off interesting in favor of easy, until one day of 16 October 2018. In the course of yet another game maintenance the game client was updated to the new and improved version. These improvements consisted of additional list filtering options (could live without, but fairly useful), slight interface reshuffle (newer one looks like a hack-job by someone with zero experience in UI/UX, but the changes are relatively minor, so oh well), some new game elements (recovery of the last copy of retired dolls - rather useful... for retards who do things first and maybe think later) and absolutely inacceptable and completely horrible interface delays in reaction to user input. Old interface in this regard was somewhat crappy, but bearable, especially as delays were masked by the inevitable for the online game network requests to the game server. Having to wait half a second because game needs to roundtrip to the AWS and sync state with the central DB is one thing. Having to wait three seconds while the game struggles to show you an empty list menu with no network activity in the background is something else entirely. Maybe in the age of websites which spike usage of multiGHz multicore CPUs when scrolling a fucking page with text in the browser and even then sometimes cannot keep up with display refresh this is considered acceptable performance by the general public. For someone with at least half a brain, however, ridiculous amount of bloat happening behind the scenes that leads to this, is shocking, to say the least.<br />
<br />
In short, incompetent monkey developers fucked up the client and then incompetent testers, if they exist at all, failed to catch this performance regression. The client was shipped and now everyone including me is stuck with it<a href="#fn3" name="fn3ref">[3]</a>. Public relations guys tried to "address the issue", even organized some questionnaires to check if only some part of the userbase was affected and I'm still not sure whether it is so, given how many active players are there (more than 50000) and how few complained (50-100 or so). One month later it looks like they didn't even start fixing the bug. The end result for me was that leveling efficiency of my dolls dropped 30% because every screen transition was taking so much longer, enjoyment dropped even more because nobody likes being fed horseshit and I stopped actively playing the game as intended and started playing with the game. I poked it a little, found it was written in C#/Unity with hotpatches in Lua, avoided cracking protection on encrypted main dll by finding someone's deobfuscated version, decompiled and located the relevant parts and made a scraper. After fixing some stupid url copypaste errors it even worked. Then I found out the server was sharded and spent a couple more evenings rewriting the scraper into the multithreaded version. Half a day before 01 Nov 2018 I started full capture and it has been running ever since on my PC. If I had another host I'd add fault-tolerance, but so far I only lost an hour of data due to power outage.<br />
<br />
<span style="white-space: pre;"> </span><b>Part 2, gathering data.</b><br />
It's funny, but developers who wrote the backend part were actually sane. Results are sent to the client as JSON, so I didn't need to bother with pulling custom parser out of game's insides. Result is an array of crafts with various information: user name, user level, item id which is converted into item name and picture by the game, mp/ammo/mre/part (4 resources) input, "crafting level" (relevant for heavy construction) and amount of crafting contracts and cores spent (which is strictly derived from crafting level, so I don't know why they put it here). All of that is good already, but developers were super generous and added some extras which don't appear in the game UI, but help us immensely (I'll show just a bit later why): crafting time as unix epoch, item unique id in the shard DB, user UID, dev_type (again, parameter derived from crafting level) and build slot (we have 2 by default and can add more, up to 8).<br />
<br />
It is not enough to capture some data then throw it at statistics and expect informative results. As the saying goes, Garbage In = Garbage Out. You need to understand what you're doing to not do something silly without realizing. My capture idea was not new. The game english server opened in May 2018, while the chinese one was running for more than two years already, then there are also taiwanese, hong kong and korean versions (and japanese, but it's fairly new). Chinese users also made a scraper for their game version, then captured results and published aggregate statistics. <a href="http://gfdb.baka.pw/statistician.html">Their website</a> shows that the capture process was stopped for some reason on 2018-09-22, but results up to that time are still available. Whether they are usable and to what extent is another question.<br />
<br />
First completely obvious problem with them that was noticed by several posters in various places is that new dolls are regularly added to the production pool, but their pages display only one set of aggregate results that shows no signs of being reset. Since sum of all craft probabilities must add up to 1, as you always get something and always get only one thing each craft, <i>something</i> is happening to the probabilities of old dolls, as new dolls dilute the pool. If you just concatenate old and new results you'll end with blurred garbage. Then there's an issue with rate-ups. Sometimes events happen which boost craft rates of particular rarities or particular items. That site also doesn't explain anything about it. Maybe internally they did the good thing and threw it all out (instead of doing the best thing of capturing and presenting it separately), but seeing how they dealt with doll addition... does not instill confidence. And mixing rate-up with normal rate is not a trivial uniform low-intensity noise: if, during rate-up, some obscure recipe is crafted disproportionately many times, it will continue to appear alongside more popular recipes, with the perceived [non-existent during normal times] rate that can approach boosted rate during rate-up, whatever it is.<br />
<br />
Back to the raw data. If you look at one capture result you will notice something interesting (since you don't actually have it you'll have to trust me or make your own scraper). Item unique ids are continuously decreasing, well, almost (dev_time resolution is 1 second and if several results happen in the same second they are in reverse order instead - either an oversight or with this sort order queries executed faster or nobody cared). At first. If you go down the list holes in item unique ids start to appear. I could have missed this effect if developers didn't include item unique ids and dev_time's and I didn't do sanity checks on results. "Who cares about holes?", you might say. "We just got less data from the server, we'll compensate with more time spent gathering". Not so fast. If you're curious, try to guess what could be wrong with this data and I'll carry on typing this sentence for a while, so that speed readers have enough time to stop reading, look away from the answer on the next line, collect their thoughts and think for themselves.<br />
<br />
Ok, here's the explanation. As you go further and further back in time according to a single log file, there are more and more results missing. Don't know about you, but my first thought was that the server was returning last N results from its <i>current</i> table of users' items. In other words, to save space and, much more importantly, cpu time of the DB server(s), once you retire something, it is erased from the table. Then it no longer appears in the last craft list. Oh, could it be that common items get retired relatively more often than rares? I looked at the equipment list to confirm - rows at the bottom of the list were almost exclusively top 5* rarity. If you used only one such list for your estimates you can guess how skewed results could become.<br />
<br />
In normal conditions actually that's not a very big deal. Inside the client there is a switch that allows it to resend request for fresh "last craft" list after more than 1 hour elapsed since the last request, otherwise it uses local cache (although you can always force refresh by restarting the game). Outside, the API endpoint caches list result on the server side for about 10 minutes (I should probably test it's really per-shard cache, not per-user, but common sense suggests it is). So, just ask for results every 3-5 minutes, get 100 or so new dolls (typical craft rate for the shard) in the list out of roughly 1000 (typical list length, there are no switches to ask for more or less) once it refreshes, append to your local list, wait, repeat. Now you're capturing everything you can without insider access. In doing so I achieved >0.99 capture ratio and it's possible to tell that only thanks to provided item unique ids.<br />
<br />
Wait, "in normal conditions"? Yeah, there has been one exception so far. Care to guess?<br />
<img src="https://sites.google.com/site/tlmcfiles/rate-up.png" width="100%"/>
These are your crafts [left part]. This is rate-up [maintenance pause in the middle]. These are your crafts during rate-up [right part].<br />
<br />
During rate-up, quite predictably, craft rate goes up too. For example, during the last rate-up of 13-15 Nov 2018 (which included Contender,Spitfire,Zas M21,Ribeyrolles,AEK-999) craft rate was 2 times higher than average during the second day and up to 30 times higher than average during the first hours of the first day. Many of those rapid-crafters run out of free space quickly, as they consume tens or hundreds of crafting contracts, so they have retire majority of obtained dolls immediately after getting them. On the chart blue is the visible craft rate averaged over 1-minute intervals based on count of reported crafts, red is the true craft rate averaged over 1-minute intervals based on delta in unique doll ids. The true rate should be the upper envelope of the visible rate, but as you can see in the beginning server reports can't keep up even during the very recent (relative to cache refresh) parts. All of this means that, if you look at the zoomed-in graph once again, at best only less than 1 minute out of 10 from each request is usable and first several hours should be discarded completely. Fortunately, this only affects estimates of things during the rate-up boost. Individual recipe/item rates outside of rate-up can still be derived from long periods of calm normal crafting.<br />
<br />
<span style="white-space: pre;"> </span><b>Part 3, analyzing data.</b><br />
To be added later.<br />
<br />
<span style="white-space: pre;"> </span><b>Part 4, EN GFDB.</b><br />
I'm still not certain if I want to release all these results as a public continuously updated website, similar to chinese GFDB. On one hand, letting the effort rot locally would certainly be a waste. On the other, all user-driven activity about and around the game is ultimately a free gift to the company, as it increases user engagement (therefore revenue). Given their shitty treatment of users (see the part about lag bug) right now I don't feel like gifting them anything. In case they release an update that COMPLETELY eliminates lags introduced by the 2.0221_268 client, I might change my mind, if it happens before I lose interest in the game.<br />
<br />
<br />
<br />
<a href="#fn1ref" name="fn1">[1]</a> Kids' grade dimensional analysis tells us that [USD/time] = [USD/gem] * [gem/resource] * [resource/time]. Resetting logistics takes, say, 20 seconds per logistic, and gives ~ 350-500 weighted resource (with up to 1.3 multiplier for great success). Current special ("discounted") offer is 9k mp/ammo/mre or 3k parts for 480 gems. Gems cost 80 gems/USD (without first purchase bonus, since we're talking about whales). All of this translates into equivalent rate of about 40-80 USD/hour, which is surprisingly less than I expected. If you earn more than that then it would be cheaper to buy resources directly, rather than resend logistics, under the assumption that purchase happens instantaneously (it doesn't). One advantage to buying resources is that you can perform a lot of it, while you run out of logistics to send very quickly.<br />
<br />
<a href="#fn2ref" name="fn2">[2]</a> Four input resources may be visualized as coordinates in 4-dimensional space, and true item probabilities (you aren't seriously expecting anything more than Bernoulli process here, are you?), which we attempt to estimate, as scalar fields inside the hypercube.<br />
<br />
<a href="#fn3ref" name="fn3">[3]</a> Technically I could install old client and MITM the connection, pretending to the client that it was talking to the older server version and pretending to the server that it was talking to the newer client version. In practice I'd risk being banned, or worse, open a whole new can of my own personal bugs because of that.rwxhttp://www.blogger.com/profile/11961724108899856582noreply@blogger.com6tag:blogger.com,1999:blog-8200259307795996652.post-40931289760538337632018-01-01T23:50:00.001+03:002021-06-11T20:58:38.518+03:00TLMC v.19 (2018.01.01)This is the 19th version of Touhou Lossless Music Collection torrent, current total file size ~1.75 TiB. Download and seed. If you have an older version, you can update it with <a href="https://sites.google.com/site/tlmcfiles/ren19.py">this script</a> (run the script in the torrent directory after stopping the old torrent but before starting new torrent). The script renames files/directories which had their names updated at some point in time - if you don't run this script you'll end up with some duplicates. <br/>
As always cuesheets are located in a separate 7z archive - when extracted into the torrent directory it places cuesheets in proper places. <br/>
<hr>
Main torrent (7z with cues and *.tta files): <br/>
<a href="https://sites.google.com/site/tlmcfiles/Touhou%20lossless%20music%20collection%20v.19.torrent">Touhou lossless music collection v.19</a> (<b>1.65 TiB</b> or 1 817 738 246 162 bytes). <br/>
Torrent file size is 9 604 641 bytes.<br/>
<a href="magnet:?xt=urn:btih:7F2010E66FF1542E62F9C8685C9E8421D16F289E">Magnet link.</a> <br/>
<br/>
Pictures torrent (cover/booklet scans): <br/>
<a href="https://sites.google.com/site/tlmcfiles/Touhou%20album%20image%20collection%20v.19.torrent">Touhou album image collection v.19</a> (<b>76.5 GiB</b> or 82 153 065 676 bytes). <br/>
Torrent file size is 4 582 077 bytes. <br/>
<a href="magnet:?xt=urn:btih:6FB45EF44298938FBF6D6FC017305D6A5BDE2B2A">Magnet link.</a> <br/>
<br/>
Supplementary materials (everything else - pdf booklets, bundled wallpapers, lyric texts, other extras): <br/>
<a href="https://sites.google.com/site/tlmcfiles/TLMC%20supplementary%20materials%20v.19.torrent">TLMC supplementary materials v.19</a> (<b>24.2 GiB</b> or 25 978 962 058 bytes). <br/>
Torrent file size is 2 200 879 bytes. <br/>
<a href="magnet:?xt=urn:btih:11078DCD8A4E8D092815406DE4290FCE23A877BB">Magnet link.</a> <br/>
<hr>
<br />
<a name='more'></a>
Additional links: <br/>
<a href="https://en.touhouwiki.net">Touhou wiki</a><br/>
<a href="https://thwiki.cc">Album wiki</a><br/>
<a href="http://151.80.40.155">Online player with direct downloads</a><br/>
<br/>
Fun facts: This is the largest update to date, at about 375 GB it is almost twice as big as the whole TLMC v.01. <br/>
Not so fun facts: It also took the most time. <br/>
<br/>
Special thanks to: <br/>
- friends at toholmc.com, who provided about 760 links to tta+cue albums, which is a majority of albums in this update (if you're reading this, please learn how to input all unicode characters instead of replacing them with question marks and underscores because going through ALL cue sheets fixing errors by hand is seriously tiring; also 3 albums were missing cue sheets and them being single-track is not an excuse to omit cues and 2 or 3 images were in the album archives they didn't belong to and 1 album contained a duplicate of another instead of what its name indicated and since crappy 400px covers do not replace proper scans why even bother putting them in ...). <br/>
- friends at bbs.moem.cc, who provided about 90 links to wav+cue albums (remaining 900 links are unfortunately either trash mp3, unavailable or already present in this torrent). <br/>
- several baidu users and baidu search engines which indexed their links <br/>
- all original album rippers <br/>
- all touhou doujin circles <br/>
- and you! <br/>
<br/>
A bonus for everyone who patiently waited for this release: discographies of artists every man of culture should listen to and enjoy. <br/>
<br/>
<a href="https://sites.google.com/site/tlmcfiles/HatsukiYura_2018-01-01.torrent">Hatsuki Yura complete lossless discography</a>, excluding collaboration albums (ones where she sings like 1 track out of 10; all albums listed in the Circle section on the artist site at the moment of this post are present).<br/>
Data size is <b>15.9 GiB</b> or 17 080 348 619 bytes. <br/>
<a href="magnet:?xt=urn:btih:86549AF1D3950C2310FB0B9EB3B42AA17EA43BF5">Magnet link.</a> <br/>
It comes with two relatively minor issues: <br/>
1. "HAMELN Limitedver" album has two glitches because the fileshare kept giving me the same broken file (made about 30 attempts in hopes of hitting another CDN shard, each time file hash was the same). Although I could technically try to recover it by checking for all single bitflips in each block (fortunately the archive package had zero compression), doing this with reversed specs of a closed-source format is rather annoying so I didn't bother. <br/>
2. "RAPT AQUARIUM" album was reconstructed from split tracks without the rip log, so there is some chance pre-gaps are missing. <br/>
<br/>
Cuesheets are located in a separate 7z archive, just like with TLMC. <br/>
<br/>
<b>FEEDBACK REQUESTS</b> <br/>
1. For circles with names in kana/kanji I looked at the circle websites. If they used an english version of the circle name then I used that as a spelling hint/alternate name in directory name: "[main name] alt.name". The rest I've left as-is. <br/>
What are your thoughts on adding alternate english names to all circles? <br/>
Pros: it would make navigating directory structure and referencing circle names easier for those who don't know/are bad with kana/kanji. <br/>
Cons: running rename script would be effectively mandatory, at least for that one transition, and straightforward name translation or transliteration might not mean exactly what artist intended. It would also break your existing playlists. <br/>
2. There is a large amount of recent rips which are sadly shared as separate flac tracks with no rip logs. For now they are not included in TLMC. <br/>
What are your thoughts on gluing them back into CD images and adding those to the torrent? <br/>
<br/>
<b>FAQ</b> <br/>
Q: Why TTA and not FLAC? <br/>
A: Two reasons, one historical and one technical. <br/>
Historical one is that a huge chunk of early albums were in tta, so to be consistent about the codec I converted the rest (mostly ape) into tta and then the majority of albums were also shared as tta, so less work for me.
The technical one is that I like tta better. Flac is open source and rather popular, it's the most popular lossless format on a lot of metrics, which is a big reason to adopt it. Tta is also opensource and less popular, but not completely unknown, and several of its design decisions (whether intentional or accidental) stand out to me as flat out better. These are: only one compression level and no embedded metadata support*. According to my preferences this outweighs flac's popularity. <br/>
* Mixing immutable data with metadata is a big no-no in my eyes and leads to disappointment and sad kittens. You wouldn't make a kitten sad, would you? <br/>
<br/>
Q: Why album+cue and not split tracks? <br/>
A: Audio CD has one track that encodes its entire LPCM data stream. Track breaks are the metadata content. <br/>
<br/>
Q: Why are cuesheets included as an archive instead of as is? <br/>
A: So that you would be able to edit them (for whatever reason, such as finding a mistake) and not break the whole torrent. <br/>
<br/>
Q: When will the next version get released? <br/>
A: I am searching for a better way to share TLMC and lossless doujin music in general. While bittorrent somehow manages to get job done it is (and always been) a wrong tool for this purpose. Sadly it is good enough, so very few are pressured to develop better alternatives. With that in mind... <br/>
Best case scenario: A p2p similar to what I described in my May post gets developed and the need for this aggregator torrent is gone, as all future albums that get ripped and shared appear and are available on it, immediately and permanently. <br/>
Realistic scenario: TLMC v.20 in summer-winter 2019. <br/>
Worst case scenario: Everything is eaten by grey goo. <br/>
<br/>
Q: Anything I can do to help? <br/>
A1, Easy mode: Seed. <br/>
A2, Normal mode: Compile a list of "file+cue" albums that can be downloaded on the internet but are missing in TLMC and send me links to them. "Can be downloaded" part is pretty important here. <br/>
A3, Hard mode: Buy missing albums, rip with EAC in secure mode and a proper drive read offset to single file tta+cue (or single file wav+cue if you're too lazy to run tta encoder, or single file flac+cue if you're already used to flac), then share on any reasonable host (mega preferred, mediafire is fine too; but even a bad host is better than no host at all, just make sure it doesn't require registration - these ones I drop on sight) and post a link. Even better if you can scan the booklet and covers (don't overkill like, ahem, certain someones; 600 dpi is surely enough). <br/>
Alternatively, find out how to contact nekomimi Alice and participate in toholmc group buys. <br/>
<br/>
And finally, if you liked the music, please support its authors!rwxhttp://www.blogger.com/profile/11961724108899856582noreply@blogger.com656tag:blogger.com,1999:blog-8200259307795996652.post-81528856120607547342017-05-13T18:31:00.000+03:002019-08-04T23:45:25.325+03:00What's wrong with bittorrent and what can we do about it? Vision of a next-gen p2p filesharing system.In this short writeup I review the history of filesharing services, highlight weakness of currently most widely-used filesharing protocol (bittorrent) and propose directions for improvement.<br />
Intended audience: anyone with an interest in p2p filesharing.<br />
<br />
<a name='more'></a><br />
<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>What's wrong with bittorrent and what can we do about it?<br />
So, why does bittorent suck? Actually, by itself, it doesn't. Bittorrent is a pretty good file transfer protocol.<br />
The first widely successful p2p filesharing application was Napster, appearing in 1999. It could transfer files directly between users' computers, but the list of files was stored on the central index servers, which handled the searches. It comes as no surprise that the company behind it was sued and shut down the service two years later.<br />
Napster's weakness was centralization, its single point of failure. Next developments tried to address that. Still kind of not wanting to let go the client-server paradigm there appeared Kazaa and Edonkey network. They still had supernodes/servers, but as they were under user control one could not simply shut them all down overnight.<br />
They had a problem with leechers, though. Since Edonkey relied on users' goodwill to share their upload bandwidth and Kazaa made an extremely unwise decision of trusting the client to correctly report its contribution to the network, many chose not to bother with sharing back. The p2p solutions worked, but one could do better.<br />
Then came bittorrent. With a single conceptual change in the form of a Tit-For-Tat algorithm being a pure leecher became <i>personally unprofitable</i>. You could throttle your upstream and (on a swarm with a high leecher-to-seeder ratio) watch as no one wanted to trade with you, only occasionally gifting you some chunks during optimistic unchoking, and wait to get noticed by seeders. If, however, you granted the application a reasonable amount of upload speed, the popular download would easily saturate your pipe.<br />
However, bittorrent entirely omitted (and thus outsourced) one critically important feature of any p2p filesharing system: metadata management. Bittorrent protocol did not specify how end users should work with metadata at all, this was left out as an implementation detail. What appeared next is tracker frontend websites, public and private, both of which now exist and have their own problems.<br />
Public trackers struggle with retention, because users have no incentives, other than pure altruism, to keep sharing files they downloaded long ago. Even is you want to keep sharing, many bittorrent clients are not well-suited to handle thousands of torrents simultaneously.<br />
Private trackers more often than not reek of unfounded elitism, they are a pain to get into, and most importantly, they impose rules which unnecessarily restrict filesharing and waste an unused surplus of <i>upload</i> bandwidth.<br />
So, bittorrent is a good file transfer protocol, but it is not a filesharing solution. A filesharing solution consists of a protocol/software/framework to exchange both data and metadata, and bittorrent takes care only of the first part.<br />
<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>Vision of a next-gen p2p filesharing system.<br />
Metadata is "data about data". The bytestream (file content) which gets interpreted as a certain container type and then split and decoded into audio, video, static images or any other form of media is data. The album name, list of tracks, movie title, its release date and all other data that describes the data is metadata. As a user of any p2p network who wants to download something, first you interface with metadata provider (perform a search), and then extract pointers to the data files themselves and let the application download them.<br />
<br />
At the moment there are two models of working with metadata: centralized and distributed.<br />
<br />
The most common example of the centralized model is a bittorrent sharing site. Torrents get uploaded by users, approved by moderators under the control of site admin, get categorized and tagged using site-local metadata schema, then users visit the site, use site search functions to locate desired torrents and download them.<br />
Strengths: Centralization helps maintain quality of both data and metadata. Low quality data files are either rejected or eventually replaced with better ones, metadata is properly organized and timely updated.<br />
Weaknesses: The website itself becomes the SPOF under threats ranging from simple funding issues of site admins to harassment by state police following orders of media cartels. If the site is taken down all metadata creation effort frequently goes down the drain.<br />
<br />
Distributed model is represented by self-contained filesharing solutions such as Perfect Dark or its predecessor Share, or, even earlier, serverless (DHT) Edonkey network.<br />
Strengths: Resilience.<br />
Weaknesses: The task of supplying correct metadata is completely in the hands of hordes of end users - uploaders. Instead of a proper schema the metadata is flattened into simple filename strings with dumb regex search as the only way to query. File collections are hacked into the system through archives with all the downsides it implies. Difficult to impossible to fix incorrect metadata. Searches can be slow and incomplete.<br />
These problems of distributed-type networks, in my opinion, contributed to their lesser share today, relative to bittorrent protocol and its supporting websites.<br />
<br />
Wouldn't it be great if we could somehow combine strengths of both approaches without their weaknesses? But wait...<br />
There are two aspects to metadata in filesharing networks. One is tagging/updating/fixing errors, you can also call it "write access" or "creating/managing metadata". Another is querying/searching/browsing, call it "read access" or "using metadata".<br />
In centralized model both "read" and "write" metadata access is centralized. Centralized model is good at "creating/managing", is ok at "using" and it gets worse with size (site popularity requires money to maintain and pay for traffic/CDN and ultimately attracts law enforcement).<br />
In distributed model both "read" and "write" metadata access is distributed. Distributed model is very bad at "creating/managing", somewhat bad at "using" and is mostly indifferent to size (but higher numbers of active users make it harder to target individual users).<br />
<br />
Key insight: one should centralize "write access" and keep "read access" distributed. Details on how to create centralized write access and improve read access to the level of centralized model over distributed network are below:<br />
<br />
The role of the site admin (root administrator for a certain collection of metadata) can be performed by any user with a private-public keypair. Moderator access is granted by signing their public key with the root key, and checked by signature verification. Public keys and signatures are broadcast into the network. Anyone can generate a keypair and become a "site admin".<br />
Each user can pick an arbitrary amount of keys to trust as root keys. For those keys the user keeps a complete local copy of the associated metadata database, together with the log of all updates. Metadata updates are database changesets signed and broadcast by any user, which are then checked and signed (or rejected, by letting them expire over time) by moderators. Owner of the root key ("site administrator") monitors published updates, resolves update conflicts, imposes an ordering on updates, signs them with root key and broadcasts these updates into the network, which then get accepted by software of the users who chose to trust that root key and get added to their local database. A search would simply be a query to this local database. An equivalent of RSS bittorrent feeds would be an ability to set precise file download triggers based on contained metadata.<br />
<br />
Why is this an improvement over bittorrent with websites?<br />
- Metadata is public and secure, spread among all users. By keeping full log of changes not even root key user can maliciously erase it at will. Easy to fork if root starts slacking.<br />
- It relieves end users of local file storage micromanagement. You can stop bothering yourself with putting downloads into appropriate directories on your hard drive(s) according to your chosen criteria and 1)let the software place them automatically based on the metadata-to-directory mapping you decide or 2)let the software store files by hash and use its metadata to locate files you need instead of relying on the filesystem as an ad-hoc database (because a generic graph can do more than a tree). A FUSE module which plugs into the metadata db and provides a file view overlay is also a good idea.<br />
- It should increase retention of old files by incentivizing users to keep downloaded files available, because TFT would account for any data transferred between peers**, not just data belonging to the same file or file collection (single torrent). Curiously, "public bittorrent" could also easily do that, but client authors didn't bother with adding that change for some reason (afraid of decreasing individual swarm performance despite increases in overall network health?). "Private bittorrent" tries to do almost just that (by tracking total ratios), except it does so in an inefficient and susceptible to forgery - in other words, plainly broken - manner.<br />
**: actually, between owner groups of an end-user key. You could push a copy of your key to the seedbox, let it do most of the uploading work, and enjoy fast downloads to your home PC because your key would be recognized by the peers as a good exchange candidate.<br />
- It allows you to "backup" your downloads by keeping references to downloaded files and backing up only that (which is megabytes at most, instead of gigabytes to terabytes). You'd have to redownload files in case of a hdd crash, obviously, but at least that would be automated. Bittorrent can also kind of do that, if you keep torrent files, but after several years you might find speeds lacking.<br />
<br />
Half-baked ideas:<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>Separation of content - pure metadata vs. data links.<br />
Pure metadata is metadata describing content without any references to actual files (say, album/movie title). Data links are, as their name implies, junctions between pure metadata and the space of files ('there is an instance of "this" movie in the p2p network and it has "that" hash' kind of statement). One could argue which files are worth adding to the set of available files: for example, TLMC never includes lossy transformations of original content (for reasons), while others might find value in mp3/ogg version of TLMC and music in general. There is less disagreement about pure metadata: it is an objective statement about the state of the world, rather than personal preference. Therefore, it makes sense to separate the two.<br />
<br />
Some open problems: there is a number of difficulties I don't know how to solve at the moment. I don't know how much impact they will have on the viability of the whole idea.<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>Ease of use.<br />
Users would have to download metadata database, which could grow into multi-gigabyte range (for example, for music, 1M albums x 1K metadata/album = 1G metadata) and then keep up with all updates. This is not something every casual user would accept and tolerate, just to download a couple of songs. It is thus not unreasonable to imagine lightweight clients, which store only metadata about local downloaded data and send search requests to nodes which do store complete copy, probably in exchange for a small chunk of bandwidth credit, but this adds levels of complexity I'm uncomfortable with and, what's worse, starts dangerously pointing in direction of central search servers, unless users are explicitly made aware of personal downsides (naturally slower searches? would that be enough to deter majority?) of choosing the lightweight option.<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>Degree of coverage.<br />
Suppose there is a database of all touhou music. However, all touhou music is a subset of all doujin music, which in turn is a subset of all music. And if you go one step further, it is a subset of all media. Now, having one giant database of "all media" is clearly impractical because its schema would grow into an enormously complex beast (I might be wrong here, though), and the db itself would get huge. But if you keep the databases split at lower levels you'd either have to duplicate maintenance efforts, or you'd need regular grafts between them, or they'll just desync.<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>Post-moderation.<br />
One attractive property of a centralized system is (an optional) post-moderation. Most users are assumed not malicious, so they are allowed to post content that will be verified by moderators later. In case a fake or otherwise undesirable file is shared, their privileges might be revoked. Significant reduction in share latency on all content outweighs occasional temporary rogue files, this can be further tweaked by requiring a certain level of trust to be established before content posting rights are granted.<br />
It is unclear to me how to implement this in a distributed system, since metadata updates do not commute and there is no central point which can serve for automated conflict resolution. Maybe it's a reasonable choice to accept delays for pure metadata db and store changes signed by moderators and trusted uploaders as ephemeral updates for data link db until confirmation from root comes in.<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>Update poisoning.<br />
The network should accept all metadata update requests and store them for some time for moderators to review. An attacker could flood the network with trash requests.<br />
One option of dealing with this would be to require users who want to publish updates to perform one-time computationally intensive task to start participating. For example p2p software could require their identifying credentials to have a property of having a cryptographic hash function of their public key contain a certain number of leading zero bits. Also, nodes should limit their storage for each publisher key and keep only last something broadcasts (like square root of their previously verified updates). Turns out this particular one is not a big problem, after all.rwxhttp://www.blogger.com/profile/11961724108899856582noreply@blogger.com27tag:blogger.com,1999:blog-8200259307795996652.post-48392316959361443292015-06-30T20:50:00.002+03:002019-08-04T23:44:09.648+03:00TLMC v.18 (2015.06.30)
This is the 18th version of Touhou Lossless Music Collection torrent, current size ~1.35 TiB. Download and seed.
If you have an older version, you can update it with <a href="https://sites.google.com/site/tlmcfiles/ren18_rev2.py">this script</a>.<br />
<br />
<hr />
Main torrent (7z with cues and *.tta files):<br />
<a href="https://sites.google.com/site/tlmcfiles/Touhou%20lossless%20music%20collection%20v.18.torrent">Touhou lossless music collection v.18</a> (<b>1.35 TiB</b> or 1 482 171 635 871 bytes).<br />
Torrent file size is 7 834 455 bytes.<br />
<br />
Pictures torrent (cover/booklet scans):<br />
<a href="https://sites.google.com/site/tlmcfiles/Touhou%20album%20image%20collection%20v.18.torrent">Touhou album image collection v.18</a> (<b>45.8 GiB</b> or 49 216 533 345 bytes).<br />
Torrent file size is 3 433 336 bytes.<br />
<br />
Supplementary materials (everything else):<br />
<a href="https://sites.google.com/site/tlmcfiles/TLMC%20supplementary%20materials%20v.18.torrent">TLMC supplementary materials v.18</a> (<b>17.4 GiB</b> or 18 631 031 061 bytes).<br />
Torrent file size is 1 572 957 bytes.<br />
<hr />
<br />
Additional links:
<br />
<a href="http://en.touhouwiki.net/wiki/Touhou_Wiki">Touhou wiki</a>
<br />
<a href="http://re-persona.com/music/touhou-game-soundtracks/">Touhou 01-14.3 Game OSTs</a> (~13 GB) by rush_art
<br />
<a href="http://touhou.kuukunen.net/">Touhou online music player (flash required)</a>
<br />
<a href="http://otokei-douj.in/">Touhou mp3 DDL site</a>
<br />rwxhttp://www.blogger.com/profile/11961724108899856582noreply@blogger.com198tag:blogger.com,1999:blog-8200259307795996652.post-61668461916142012752014-11-16T18:24:00.000+03:002014-11-18T00:22:00.709+03:00May every day of your life bring you smiles and joy.
<img width=800 src="https://sites.google.com/site/tlmcfiles/2ca4c089f4e55cc2e97317962eb618a9.jpg"/><br />
Happy Birthday, Yuno!rwxhttp://www.blogger.com/profile/11961724108899856582noreply@blogger.com5tag:blogger.com,1999:blog-8200259307795996652.post-88678708546570318072014-03-30T22:30:00.001+04:002014-04-06T12:58:18.597+04:00Today we celebrate another birthday of a heavenly being.
<a href="https://yande.re/post/show/261835/"><img width=840 src="https://sites.google.com/site/tlmcfiles/yande.re%20261835_80.jpg" /></a><br />
Happy Birthday, Rei!rwxhttp://www.blogger.com/profile/11961724108899856582noreply@blogger.com5tag:blogger.com,1999:blog-8200259307795996652.post-10113006896949156122013-12-25T03:13:00.000+04:002014-02-24T04:01:55.385+04:00A gentle reminder that today is the birth date of our beautiful angel, Queen of the skies.
<a href="http://konachan.com/post/show/74358"><img src="https://sites.google.com/site/tlmcfiles/30_Konachan.com%20-%2074358%20feathers%20ikaros%20long_hair%20pink_hair%20sora_no_otoshimono%20wings.jpg" /></a><br />
Happy Birthday, Ikaros!rwxhttp://www.blogger.com/profile/11961724108899856582noreply@blogger.com5