#+hugo_base_dir: site #+hugo_section: posts #+STARTUP: logdone #+author: Valentin Boettcher * Pages ** About :PROPERTIES: :EXPORT_HUGO_SECTION: / :EXPORT_FILE_NAME: about :END: This is the web presence of Valentin Boettcher a.k.a. Hiro98. Valentin: - currently studies physics in Dresden. - codes stuff. - is a science-fiction junky and space enthusiast. - loves jamming with the saxophone. - is hooked on LISP and Clojure. - appreciates C++. - likes his [[https://blog.splitkb.com/blog/introducing-the-kyria][Kyria]]. You can check out my [[https://github.com/vale981][Github]], [[https://gitlab.com/vale9811/][Gitlab]] and [[https://invent.kde.org/vboettcher/][KDE Invent]] profiles for most of my projects as I publish the source to most things I produce. I've got a [[file:static/docs/cv_ger.pdf][CV]] in German. My usual internet/user/nick name is /Hiro/ or /Hiro98/ after Hiro Protagonist from Neal Stephenson's novel "[[https://en.wikipedia.org/wiki/Snow_Crash][Snow Crash]]", who is usually depicted in profile pictures like the following one. #+DOWNLOADED: screenshot @ 2021-08-01 15:14:54 #+CAPTION: By [[http://www.benzilla.com/?p=4209][Ben Towle]]. [[file:static/images/Pages/2021-08-01_15-14-54_screenshot.png]] *** Things Made - The source of this website lives [[https://github.com/vale981/website][here]]. - I've written a bachelor's thesis about [[https://git.io/JBPZg][Monte Carlo Event Generators]]. I even built my own in python. - A simple, easy-to-use and firewall penetrating [[https://gitlab.com/vale9811/doccam-pi][control software]] for streaming stuff from ~rtmp~ cams to youtube. - Along with a commilitone I've written some lecture notes about [[https://gitlab.hrz.tu-chemnitz.de/strunz/skript-quanteninformation][Quantum Information]]. - The website of [[https://simunova.com/][SimuNova]]. - The software behind [[https://klausurnoug.at][klausurnoug.at]]. - A [[https://protagon.space/stuff/neutrino_oscillations/][visualization of neutrino oscillations]]. - A [[https://github.com/vale981/SecondaryValue][python library]] to calculate the gaussian error propagation. - A [[https://git.io/JBPZX][small hack]] to scrape and organize physics courses at the TU-Dresden. [[https://protagon.space/stuff/vertiefungs_scraper/][See it in action]]. - A script [[https://github.com/vale981/wunderlist-to-org][to convert the wunderlist export to org-mode.]] - A script to create an arch linux image with [[https://github.com/vale981/archiso-bcachefs][bcachefs]] support. - I've made the [[https://sherpa-team.gitlab.io/][website]] and some of the continuous integration for the SHERPA project. *** Activities - I am the admin and the most enthusiastic user of the [[https://physik.protagon.space][TU-Dresden Physics Forum]]. - I am an occasional contributor to [[https://invent.kde.org/education/kstars][KStars]]. As of late I have participated in the Google Summer of Code to implement a better Deep-Sky-Object backend. I also implemented and now maintain the [[https://invent.kde.org/vboettcher/kstars-catalogs][KStars DSO Catalog Repository]]. - Sometimes I release some noises on [[https://soundcloud.com/the_dj_c][Soundcloud]] and [[https://afa-music.bandcamp.com/][Bandcamp]] under the name "Abschreibung für Abnutzung". - I am also the maintainer of the [[https://www.doc.govt.nz/nature/][RolaCam]] backend which is also used by the [[https://www.urbanwildlifetrust.org/portfolio/live-cam/][Urban Wildlife Trust]] for their [[https://www.youtube.com/channel/UCLizlM6gpaVHTKPo7spoqlA][Wildcams]]. ** Contact :PROPERTIES: :EXPORT_HUGO_SECTION: / :EXPORT_FILE_NAME: contact :END: Just [mail](mailto:valentin@boettcher.cf) me. Spambots not welcome :P. I'm on [dev.to](https://dev.to/hiro98). Here is my ~GPG~ public key. #+begin_src -----BEGIN PGP PUBLIC KEY BLOCK----- mQENBFsw8ywBCADP+YJLyntBCNgld21x2FG811w2VFLizjrYRakgIwqSCEYyeUd0 jXhVi8SzsPEQlwPEUtDgLQekRvsDXhT2mW4jq5g46qX1OSrzAh3H1R0e9dAVNC59 fNGDOFVEKmNdPhSOprwTMHQLsebo13gc5xm5SDSGx780xPGvRHMkPyQdQw6mufTZ 7ejbkAnsHoZKANhfu68O6GN71Rpogp4u6o/ac4TpzBNb+zuNL2NCckkf3oT9Lokb r/I0KsV0xAlbbmth4vMFeKmhlgR/TYWNqNe+Lq9tIbyX780gAzEItdgRJFT6LEjd veiAe+iWw/EJPZy9+qenoCGns//zagov1iMFABEBAAG0NVZhbGVudGluIEJvZXR0 Y2hlciAoTWFpbiBFbWFpbCkgPGhpcm9AcHJvdGFnb24uc3BhY2U+iQFOBBMBCAA4 AhsDBQsJCAcCBhUKCQgLAgQWAgMBAh4BAheAFiEERamZF1eM1imfX7W0wi1N5Nez LRkFAl5mg1IACgkQwi1N5NezLRnLOAf/RRUjZzVOu+NMShk2G+fEodNTem8pD5r1 Up+qEdywyPlV87UtK2y2Q4CB4mwuqJV7OwgQuCMVOJhSGstpQ6bQcl+VZ1p9dMzr TVqcaHWFexLp0+h/S6loGGySh0txvzYkTLfAih3osXmQvvwNjw8BggT7wWztS9q2 cmvZ1sT3TP5Vq67WyBW6We9a6R5hw9lb+IOQct2dJkskxhI6CHyZdKXvxUMk0mE4 2uf9+lbFzBTJqs+VT5TfLnTcA6Vhu6ke50mSCuMkwKEjER/lmNCx6TuSYxuVGczi hDfs4RDzpewMIkkoZK6YBtKvyIcyY5lBzmmgNJmmVbmCgFmQiEgN1IkBVAQTAQgA PgIbAwULCQgHAgYVCgkICwIEFgIDAQIeAQIXgBYhBEWpmRdXjNYpn1+1tMItTeTX sy0ZBQJg1fgxBQkJZ2wAAAoJEMItTeTXsy0Zgk4H/1BVJKhLLwnj5iWSFk+2OliU DmFhkWjNWZpOCvaBSxNQNkod+9jjQ+Rg0+thRyLm6oZOg4g/iluslds1Hxf++pxz /HJdJ7/oQ/BG3RZrh2Lc2BeptVntFoGR/K0VoJYbwjOXP8B8lnGJZEX6/QnT9Fkq ZFrm+8CgYhKradeTl30LYA5TBCT+BrtzervHv6CrmN6WcROdwYHj+RHBXauswG8k eeoDVXLpg1XiMcWB5dhiVQNxcoAiOL9k5Z6xSR6Zgd5ediXBvbL3k3nsu/ZtWeR3 7dLFnba2zi2IGDL48fchgFSWWM8SemKeQ9wiPb+no+uEFDWmwa/+cAm54DFeJ420 KlZhbGVudGluIEJvZXR0Y2hlciA8dmFsZW50aW5AYm9ldHRjaGVyLmNmPokBTgQT AQgAOAIbAwULCQgHAgYVCgkICwIEFgIDAQIeAQIXgBYhBEWpmRdXjNYpn1+1tMIt TeTXsy0ZBQJeZoNSAAoJEMItTeTXsy0ZYv8H/jdOvmnXV2Og2vEKJsoPDgvfzqww EopE8j6CADBdyKt+51CgGkccCj7SzCG80qUMNJAeKU4c5nN7OVGJ+b3hNd2G0f8j V5mRlXNIl6m+7MmflUph9PKMBRlezUTA0enBIJIvxe5l8M2dp4zO09HGtoexiVcg FX3u4DsKy2BT8c3scfKcEQXFQV+zKLBm4ko43LSEEtTS2JOCyTNorTkS6b0JBN6N Opmbb0hnx45+731vuz0/F3OsLbPJ4objiLVoM1RaY0tx0z4Kb5J9d+9y0eHAtxt/ MhemqbNXPsFZFHW2mLxEtK+dZuCeCyh+qBcFUWvPCkXk0X2bPcBLJwabJYKJAVQE EwEIAD4CGwMFCwkIBwIGFQoJCAsCBBYCAwECHgECF4AWIQRFqZkXV4zWKZ9ftbTC LU3k17MtGQUCYNX4LAUJCWdsAAAKCRDCLU3k17MtGe0eB/4rIF+j316MEdBE1/sg 8yNMM/lnYh3BBtF0WmR0NVZA5IFMJXZg029fqRtkcUSkro21XIXkMkmz5hmG2GHt 9i6/Or/irfhwZKLPBeN7KT7TJkLUNP6V2dmIF4Juz0+KIE2aBFg3zDLA6CrTCDig I7GE/8adb40SXC0XLOdLnxO0SUr1VL2Hyi1Jw6AgroYynTbAzWpYP+JkSPW1voUl /TfpPxMvgRsJS10XTQUNdYu54r/ByM/tYb7f4+KLaUNTW8zlViS8niNaASRe3e/2 5Qn/9aeytRlzZUi3UZ+tb5TrTlcNdpER16GPcb63ABfedADyZXd8NNafLDuRf/bF KR5juQENBFsw8ywBCAC4LcWjxCONetL5q33xGMLJbGU49FkMT1qpwiLTgBTG3jy+ /0TbNYnC4iDulUm0yardjPHmYLwswnzzzYWSvbpE+cKJdULw37+R0f+28eyN53dY +XZ6yb6KiFAjQCwcSaLmB7a4YIEOqpz6ake7qiuhJ6Lx5hEfp42o4kR2QxugiMqe Z+8IwhfLM+0AlP+rqdJdkxvw+dXW0C/F9hZcRAgRKZ1N+eQU8uUME5QXI8grhtd9 ThLFCo3dl6a+dI/TkRpYLVR9c4obcocUtTnzOl4L/o5FedDq020D9+MdXqbRQVTs dF2/jTVbBwhDcO6Vi46OrcFvTQ7d+TaOhuhsshpVABEBAAGJATwEGAEIACYWIQRF qZkXV4zWKZ9ftbTCLU3k17MtGQUCWzDzLAIbDAUJA8JnAAAKCRDCLU3k17MtGVe/ CAC+lp2MhdN7Enr+RE8qDz0s4090uDVI8mQWi663NRCd1uUHrQTju1dqA0MEjYFG AVhIpnHcjroWMyGNFQphs7ekLmBqsHMf5PYOtGdqKbI+y0TCfQ+GXF9tUK9a8Xp3 EwLHgGA0SaeyEZj9TwmsoetcYtZA3FNiCy2QEzsNL0dpytZFPktvR3I52eE3cxYw 9gI9x9/Z+n0vyQGL1o1CsB/z2Vh+g8QfINHHfnX1PQp3Jjbg/uhjw5mZpn6+i7/k UC9VICpHoT3U2xQHrDjCol8nvH4PSfpyPMr1is7S5I05Q1xja2GQgt5qmtpGyx4K AKTcpWc4R+XgmwwIF5NWaadPiQE8BBgBCAAmAhsMFiEERamZF1eM1imfX7W0wi1N 5NezLRkFAmDV+E0FCQlnbCEACgkQwi1N5NezLRmRQAgAuItAuBZYFUQb1cOxeKQY V3wETwd9TduZUAEUQzZQhNcfoOJhAVnOrKqUpvP11cVjxXD2ox8GaPi4NXl5BPa9 EEfnRUJQ28R9JQmDNqWrCcYjNGtuw1JwnsJKanSgV+L/8MJF5DxfqaiAKfq+t0By 3a/7SRa0X6LxciLn/uoqdy49/L223oe2InC3PbiJz3RnIOZGLTvCeYEjumZcfUx+ iM4Kzj7zD0tbH9fvboexyacp8034qlD48LE9/N9rWkvLYUfm0DiIkyt8wCUHB1r+ PQw8+Q2mSJ4JBksco0BsL96cH7LhL9RBB+UkNjxiBYaPY2vxhLImWPTCY39yA4wM I7kBDQRfGCGvAQgAsCHAvggPmnwaKV4lWhAm8+dI92iYzZfkNX54jy92UceFG7p2 9bQOnlG2SMWilIMTI9X9+oeaDx41mlAiTQrWnQ5udum3x3UYCAoQXhRZsxFjO1ob 1Ycc0K0GT0RuQrggFay7s39hpTGI5d8lYBjcLJ3oepUCe4nEvIn5rhvypbUiRBo/ +G0g9ysKKT9ehDDdp0qMxM8kRJi215jjctxFSGyvXerbb77rMkKYeVp0U58SA7WY 9yKv4tn1nBYVaTNPMwHwt17MrIz5fSZjIexl9lUpoHGEDNu+6QPdSkfhrc99Km2N 3wC5lNDzyjYifox6kzEtGmMrmIHjDiIysA9N7QARAQABiQE2BBgBCAAgFiEERamZ F1eM1imfX7W0wi1N5NezLRkFAl8YIa8CGwwACgkQwi1N5NezLRl1vQgAiES+jtp7 +bpW+rdUNGbCJAS28gQpn9Bkt54c+WyDEZbMST1cJgyuLX/0Zf8xu2Bwwxb2LYSk J7N83gDsXie0zmD3NoHE27FPEGIsVHoo0qcgcg8MY66O6ZRTr/njMF+DGPqV7FtM x5x9qfM7YMZ+3S49DmGgHRBAELGuyqfvLiTM5unlnDDA9N755VCHZ5optOcuguMY y4IhPexft/6cRBfLPH2EqpoDVuKcQDr0Y1OKinsgmzSB8qUYt6SqWTN51WfhQMtz Ejh3MC58ExFWmgOE2cS++8vk7ntprYEeeSesoXPXvXW8AvfTrjnapIg2kjtuDRBH e6q7By8xuUSSOrkDLgRfvmUSEQgAjNMfeXd6jy310NXYQqw9zcE71ubaLs7drlmJ lrp+vEDfAq351BpbDtE/AVWrMujccFVx/fDJQoipXi+8dsJswZT152NNXfEceeaU 56CUKXvlOfmDjqeSehI6/L9g7sXH8hSpngyx6CmBViP4MQPwwlwrb+N52l8ub1tn nJJf7sgBO+SnRpKa5ZRHx6YHHsQ5vYwzowwR6iK0+6NojNzlVp+8F5qdVPvygEF4 eOz1e2dmKgMEjzvaf/Z64WYGQeKsFXreRjskOuMMmDjJ/6sduOjDvrHZW04jRNFZ rwZbaE+iVv1LMxTxV5lus35lum/UUuShH6EI/fWAs/8HLe8IBwEAulBzs2L/oLw/ tw1gjL1D2PUDzaTaNl0LpSVzdB5CbNEH/idU1SSvv8Vf7uVnAnxJXQUuLnDuz3Pq uym4TvoWRM/OKd/uj+6d+BeoriUDFFuRhNJcvNLyDfU/9LOZMbxjQer+MYc/aNsd PUUlJeSgkc7pGdToC2/2IqTntb+ZYRUwUDAGMElxTARIMNcASaIgVcLwkwaeX4aN DzOxvRyh47MINVXUMfqEuDXZYEQdVXVwQQQQl3cD4q6WEiZLOhmyzIq4DTyTA/1B OExM/HvJgeczIOtGMfO3ZyLYQ7KpK1/1ZWMVV7tl1ZoXUL4JdgORqNIrbY3qqEQu riPp0m4ypKqmj7W0FtYut5TDu72O8w4hb4+3W1qMjWIrW32h0KuLHT4H/05fwzpc e56djf88QTt8C4CUx6Anks/+RmPtyNFHylQ6mgX1CfntrD4Mk4N+qRP9iZRHZuxR AkzVcIHs2p/FX5/83pyR5a3krsJvVxoeER5VhoIzZedRvwqGiAn6sA5uSPa9IIBc 6oefAsjleSHZlwC5tQ3zdtFidI3xnV65myppv5uMo73GZFM28/RmSCJgnAX6QxfL /EAXlgxa7gVWbJw6b+QAMPvH58+gm8z3rN9CXIb6k6GBKoXcnEH7imTXQ0ZhJRA5 DWu0FCziVaJ7WgXdeGRHSjd7AqAaDqUWi8DseFfBVpCX9r6nMJY1WRv8iLIb3d+U 7LS3EBh18M9PBWmJAa0EGAEIACAWIQRFqZkXV4zWKZ9ftbTCLU3k17MtGQUCX75l EgIbAgCBCRDCLU3k17MtGXYgBBkRCAAdFiEEOsdCeTthkZeWIuEa4DThK3r1as4F Al++ZRIACgkQ4DThK3r1as5azAD+KqMX5KcfcL4ibM0zpCUoC8ncFwhC1rgf263F uqAt08YA/iOIUT9TsoW4Y+e/Zx9jpBS5w8byg5tPfGusiaLG35TO05UH/2vYKVRQ URFIsxpllAIE0fCDkgVTrACYfW1GXVF2Z+2JZ9iRXw//zK3HXTPTKjlmkob+EK+5 AmrRoGuN9y4L5FEpLuycXO6u6KAQD+0fN0GUPFxzwf8kUSfZbaRWIJKEsDd/NSgJ Cu4UMICGz5Q6kzkBtRkePlhgrS6+PeJHJ0C6P1T/qVwvI/eIcPRqgeJOndG9BtsV 5/QO/rUXtMoLUVHoFANr3JNuFGZpZ0xnQEd2QGAxOU4EfQ0djt1a/YNp3qgBbU3f z9SvuganEEyWovdM1ojmJxkU7jPa6SztIaIUKsl6Zn7d8R7fJJSP3po38yuGcF+m 4gcGTSpBB5xsqnc= =x6uy -----END PGP PUBLIC KEY BLOCK----- #+end_src ** Impressum :PROPERTIES: :EXPORT_HUGO_SECTION: / :EXPORT_FILE_NAME: DSGVO :END: Verantwortlich für den Inhalt nach § 55 Abs. 2 RStV: #+begin_src Valentin Boettcher Raecknitzhoehe 15 01217 DD Kontakt: E-Mail: hiro at protagon dot space #+end_src * Blog ** KDE :@KDE: *** KDE GSOC: Intro :GSOC: :PROPERTIES: :EXPORT_FILE_NAME: gsoc_intro :EXPORT_DATE: [2021-06-27 15:00] :END: Hi folks, talking to you over the interwebs is Valentin Boettcher who is overhauling the Deep Sky Object (DSO) system in the KStars Desktop Planetarium for the Google Summer of Code anno domini 2021. This is the first post in a series and rather late in the coming, so let's get right to it. I'm currently studying for a master’s degree in physics at the TU-Dresden in, you've guessed it correctly, the beautiful city of Dresden (Germany). In Germany, we do have two study terms per year and the summer term usually coincides neatly with the GSOC so that I couldn't participate in past years. This time around however, my schedule was finally sparse enough for me to have a go at it, and here we are :). My first contact with KStars development was back in 2017 while I spent a year in New Zealand and had a lot of time at hand. My reasoning was, that I could learn mathematics and physics in UNI and should funnel my enthusiasm into familiarizing myself with software development and the open source software community. I promptly wiped my hackintosh laptop to put Linux with KDE on it[^3]. After reading ESR's famous ["How To Become A Hacker"](http://www.catb.org/~esr/faqs/hacker-howto.html), I followed the advice given therein, which was to find an open source project and start hacking on it. I already liked KDE and space, so KStars was in the center of the Venn-diagram :P. I went ahead and busied myself with one of the junior jobs listed on the KStars web-site[^2]. I quickly found that I liked figuring out how stuff in KStars worked and also got in contact with my mentor Jasem Mutlaq who was always available to answer questions and endure my barrage of instant messages on matrix :P. My second job was to draw comets a tail and learned that it is wise to do some code archaeology before going ahead and implementing functionality that is already present. From there on I contributed more or less regularly when I found the time in my semester breaks. Now, finally, let's talk a wee bit about the actual GSOC project. In KStars, everything that isn't a Star or an object in our solar system, an asteroid, a satellite or a comet (I'm sure I forgot something) is a deep sky object (DSO). Prominent members of the DSO caste are galaxies (think M31, Andromeda), asterisms and nebulae. Of course there are a plethora of catalogs for specific types of DSOs (for example, Lynds Catalog of Dark Nebulae) as well as compilations like the New General Catalogue. The system for handling those catalogs in KStars has grown rather "organically" and is now a tangle between databases, CSV files and special case implementations. Many catalogs were mentioned explicitly in the code, making it hard to extend and generalize. Also, the sources of the catalogs and methods how they were transformed into the KStars format were inhomogeneous and hard to reproduce, making deduplication almost impossible. Finally, KStars just loaded all the DSOs into memory and computed their position on the virtual sky for every draw cycle, which made all too large catalogs infeasible. My task is now (and has been since the beginning of June) to implement a unified catalog format which can be loaded into a central database and supports deduplication. Furthermore, taking inspiration from the handling of star catalogs in KStars, the objects should be trixel[^1] indexed and cached in and out of memory (but only for large catalogs). Finally, it would be very desirable to make the creation/compilation of the catalogs reproducible and easily extendable to facilitate future maintenance. This sounds like a big heap of stuff to get done and in the next post I will be detailing how it's going so far :). Cheers, Valentin [^1]: In KStars the sky is subdivided into triangular pixels "Trixels". Assigning each object to a trixel makes it efficient to retrieve all objects from a certain part of the sky. [^2]: which had to do with figuring out why some faint asteroids where missing [^3]: which I knew from my school time when I used it on my netbook because there was a cool neon "Hacker" theme for it :P *** KDE GSOC: Community Bonding and First Coding Period (May 17 - July 11) :GSOC: :PROPERTIES: :EXPORT_FILE_NAME: gsoc_1 :EXPORT_DATE: [2021-07-11 15:00] :END: Of course the task I described in the [[*KDE GSOC: Intro][last post]] looks and is quite monumental. That is why I laid some of the groundwork for my GSOC beforehand (in the actual German semester breaks). This work continued in the community bonding and first coding period and will therefore be described here. But first I want to thank my mentor Jasem Mutlaq for his support, his patience with me and his nerves of steel. My mood levels were somewhat similar to a huge-amplitude sine wave those last weeks. Now to the meat... I began by studying the existing deep sky object implementation in KStars to identify what structure the new catalogs should have and what the smallest irreducible core of functionality was I could replace to make integration easier. I discovered that the catalogs were a mix of SQL databases and text files, somehow loaded at startup and then appended to some linked list. There was some deduplication implemented but like most DSO code it was oddly catalog specific. Especially the Messier, IC and NGC catalogs were often mentioned in the code. Also the explicit distinction between stars and DSOs made writing general code complicated but I found a consistent set of data fields shared by all catalog objects which all admitted sane defaults. It wasn't bad code or anything like that. Just the product of "organic groth" with many thing I wanted already present in some way but somewhat all over the place. I admit that I studied the code just enough to find out what exactly I had to replace and maybe I could have reused more of the existing code but I've picked this specific path in the multiverse, so let's get on with it. Just a shout out to all who did previous work on the DSO code among whom are, just to name a few, Jason Harris, Rishab Arora, Thomas Kabelmann and Akarsh Simha. With this knowledge I was able to go forward and devise a concrete plan for implementing the new DSO system. First of all, albeit I would love to use ~std::variant~ and some kind of entity component system for the different DSO types I settled with a one-for-all type for deep sky objects. The primary reason for this was, that KStars uses ~C++14~ which lacks variants (and the extremely useful ~std::optional~). Furthermore the DSOs all share common structure, so this was just the simpler and thus preferable option. The second design decision was not to load all of the DSOs into memory, but instead to take inspiration from the deep star catalogs. For one they are dynamically loaded from a special trixel indexed format so this already was within the formulated goals of the endeavor. On the other hand the notion of having "canonical" copies of catalog objects in memory and syncing their mutation with the database system seemed overly complicated. The catalog database should be the single source of truth and not the (ephemeral) memory of KStars. When a specific object is needed, it should just be retrieved from the database locally in the code instead of searching some in memory list in KStars or shooting around with pointers. This notion is somewhat at odds with how things were and are done in KStars which created some interesting problems later on as we shall see. These ideas somewhat dictated the rest of the plan which I (for the first time in my programming career) completely wrote down in advance. The heart of it all is the database manager which abstracts maintaining, reading from and writing to the database. As always one should justify the creation of a special data type. In this case it was the requirement that the database access should be painless and could be handled locally anywhere in the code just by creating another instance of the database manager. The manager should handle retrieving objects and catalog meta information as well as importing, editing and exporting catalogs. The structure of the database itself was another point of consideration. Naturally each catalog should have its own table. But how should deduplication work? The method I settled on is really quite simple. Each object gets a (relatively stable) hash that is calculated from some of its properties which is henceforth called the ID. When two objects (from different catalogs or otherwise) are the same _physical_ object, then they will both be assigned the same object id (OID) which is just the ID of the object in the "oldest" catalog (with the lowest catalog id), trying to make it stable under the introduction of new catalogs. Additionally each catalog is assigned a priority value which is just a real number (conventionally between zero and one). When loading objects from the database into KStars and there are multiple objects with the same OID only the one from the catalog with the highest priority will be loaded. This simple mechanism should cover the requirements of KStars quite well and is relatively easy to implement. There I ran into an issue that demanded some research and table in the database. The simplest option would be just to create a benchmarking. Remember that each catalog is represented by its own so-called view, a dynamic "virtual" table that combines all the catalog into one homogeneous table. SQL magic could automatically perform the deduplication algorithm outline above and everything would be fine and dandy. However, benchmarking revealed that actually writing the view into its own table, henceforth called the master catalog/table, increased the performance quite considerably, enough so to justify the increased complexity in the implementation. And then I discovered SQL indexes. A gift from the heavens! They increased performance on loading objects in a trixel from the master catalog roughly threefold and I was sold on the master catalog approach. So to summarize it all; a deduplicated view of the combined catalog tables is being created and then written into the master table. This has to be done for every modification of the catalogs but is relatively fast (just not fast enough to be done 20 times per second). Later experimentation showed that this approach could accommodate catalogs up to a million objects in size. I also created a catalog file format, which is just an sqlite database file with the application id set to a special value with almost the same structure as the catalog database proper. The application id enables KStars to check if the database is really a catalog file and not to rely just on the structure of the contained database for that. In the future the ~file~ command and other utilities like file managers could be made aware of this special application id to recognize the catalog files. We will leave it this level of detail for now. For more details please refer to my [[https://protagon.space/stuff/kstars_cleaned.org][notes]]. Of course the operations on catalogs have to somehow be accessible in the GUI of KStars so this was another point of action. Before that however the glue between the database manager and the usual sky composite system had to be implemented. In KStars different types of objects (Stars, Comets, Asteroids, etc. pp.) all are implemented as components with a unified interface. These components provide methods for loading and drawing objects, as well as utilities to find objects near a certain point on the sky and similar things. The loading and drawing part was relatively simple to implement. The drawing code could be straight up reused from the old implementation and the loading was essentially covered by the database manager but with a twist. To support very large catalogs it would be desirable to only have objects in memory which are currently visible. Thus a LRU cache was implemented with the trixel id, which essentially labels a portion of the sky, as key. This cache is fully unit tested and relies completely on standard library containers so not a single pointer appears in the code.[fn:1] As an added bonus, the cache is completely transparent by default and only takes effect if configured to so and therefore includes the typical use case of comparatively small catalogs up to ten-thousands of objects. But here the culture clash between the new DSO implementation and the traditional KStars way of things became apparent. In many places KStars expects pointers to so called ~SkyObjects~ with no real clue as to where they are actually stored and how their memory is managed and with the implication that the object is expected to live forever. Well, the DSOs from the catalogs aren't supposed to be kept around forever and thus a compromise is in order. So whenever a pointer to an object is required, it is inserted into a linked list[fn:2] in a hash table with the trixel as index or is taken from there if it's already present. I hope that we can eventually transition away from raw pointers and manage life time either explicitly or with smart pointers. With this done and basic drawing working I went on to implement a basic GUI for catalog management[fn:3]. I also wrote unit tests for the database functionality which proved itself as very useful later on. After that I couldn't delay anymore. Back when I implemented the component for the new DSOs I went as far as getting it to compile and not much further[fn:4]. Now I had to go around and find out what broke. A lot broke and I did not find all of it until the big merge :P. A rather interesting source of work happened to be the way metadata like observation notes and image links were stored. They came from a text file and then were loaded into the sky objects at startup and somehow synchronized with the text files on mutation. This, of course, played not well with the new DSOs as they were ephemeral. So I replaced the whole shebang with a hash table which incidentally improved startup performance. The rest of the integration work was similarly interesting and continues today. I will not go into it further but feel free to look at the KStars commit history. Just yesterday I added a feature back in that I had axed accidentally to the dismay of its original author. That showed me that I am not entitled to judge the merit of individual features and whether they could be sacrificed for the "greater good". The answer is: They cannot! Another lesson I've learned is, that too much magic just ain't no good. I had created a variadic template wrapper for the ~QSqlQuery~ type for syntactical convenience and shot myself in the food with it. It ended up obscuring an error message and prevented me from reproducing a crash that users on certain platforms were experiencing. After a not-so-great couple of days I, with the help of two kind people, finally found the lowest common denominator of the problem: an old, but still supported version of QT which bundled an old version of sqlite which in turn did not support the ~NULLS FIRST~ directive that I was using. Turtles all the way down. Although I tested all my changes on KDE Neon (I am on NixOS primarily) the wise thing would have been to develop or at least test everything with an older QT version from the get go. Also, although I had put in version checking into the database code, I didn't provide a mechanism for upgrading the database format to new versions. This I now remedied by introducing a simple mechanism that applies database modifications successively for each version upgrade. So if I go from version two to version four it will be upgraded from version two to three and then to four which I understand is the way those things are usually done. Now, I did do at least some "constructive" work, adding a (admittedly ugly) CSV importer so that users can import arbitrary CSV-ish catalogs. The greater chunk however I will cover next week: The python catalog package tooling with continuous integration and deduplication. The catalogs churned out by that framework are then installed via the ~KNewStuff~ framework. I discovered two interesting bugs in this framework because KStars seems to be almost the only program using the framework in this specific way. If you made it this far, I applaud and thank you for your endurance. See you next time. Cheers, Valentin P.S. Currently I am working on documenting both the new DSO GUI and the python tooling. I hope eventually they will pass the "noob test" :P. But, as you may have recognized above, I am not the best explainer. ***** Footnotes [fn:4] I really appreciate c++ as a compiled language. [fn:3] See the KStars Handbook. [fn:2] References to objects in linked lists are stable. [fn:1] As a matter of fact, I set out with the goal not to do any manual memory management and not to use a single pointer in the new code. I have been successful thus far if you would be so lenient not to count glue code for legacy KStars systems. ** Small Insights :@Tricks: *** How to use the Systemd userspace DBus API on Traivis-CI :DBUS:CI: :PROPERTIES: :EXPORT_FILE_NAME: sysduser :EXPORT_DATE: [2020-07-11 14:00] :END: I am currently working on a project which involves talking to the ~systemd~ userspace session via the session ~dbus~ instance. After some fiddling around and enabling debug mode on travis via the excellent user support, I came up with the following. Travis uses VMs that run ~ubuntu~ which comes with ~systemd~. To enable the userspace ~dbus~ session, one has to install the ~dbus-user-session~ package. After the installation, it has to be activated through ~systemctl --user start dbus~. Furthermore one has to set the ~DBUS_SESSION_BUS_ADDRESS~ environment variable through ~export DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/$(id -u)/bus~. TL;DR #+begin_src yaml script: - sudo apt update - sudo apt install dbus-user-session - systemctl --user start dbus - export DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/$(id -u)/bus #+end_src *** Fixing Linux Dualboot: Reinstalling the Windows EFI Bootloader Files :PROPERTIES: :EXPORT_FILE_NAME: dualboot :EXPORT_DATE: [2020-07-11 15:00] :END: Note to my future self :). Reloading my Linux install after a pretty radical 'nuke and pave' I had to get my Windows dualboot back to work. There are a thousand guides on how to do that, but I'll add another one in case your setup is similar to mine. I have installed windows on a separate drive and Linux on my main drive, along with the efi partition. Don't follow this guide blindly. Think about every step you take, because you can seriously mess up your system :). With that out of the way, the things you have to do are: 1. Boot a windows install medium. 2. Choose your language and enter the 'repair options'. 3. Go to advanced and select 'command line'. 4. To mount the efi partition type diskpart and in diskpart then type list volume. A list of volumes will be printed and one of them the efi partition (usually around 500mb ). Select this partition (select volume ~[number]~) and assign a drive letter (~X~ is the drive letter you assign). 5. Check where your windows partition is mounted. The diskpart list volume output will probably include it. I will assume that it is volume ~C~. Exit diskart with ~exit~. 6. To finally install the boot files type the command ~bcdboot c:\windows /s x:~. This will generate boot files based on ~c:\windows~ and install them on the partition with the letter ~X~. Thats it, you can reboot now. You may have to reconfigure grub (or whatever loader you use). On arch-linux, make sure you have os-prober installed :). *** Installing without Fear :PROPERTIES: :EXPORT_FILE_NAME: inst_without_fear :EXPORT_DATE: [2020-09-16 15:00] :END: Note to self: If you want to make sure some nice GNU/Linux installer does not touch certain drives just run ~echo 1 > /sys/block/sdX/device/delete~ in a **root** shell and the drive will vanish from the system. Shamelessly stolen from: https://askubuntu.com/questions/554398/how-do-i-permanently-disable-hard-drives * Local Vars # Local Variables: # eval: (org-hugo-auto-export-mode) # org-download-image-dir: "./static/images" # org-download-heading-lvl: 3 # End: