Planet SciPy
Writing docs is not just writing docs
This blogpost was originally published on the Quansight Labs website.
I joined the Spyder team almost two years ago, and I never thought I was going to end up working on docs. Six months ago I started a project with CAM Gerlach and Carlos Cordoba to improve Spyder’s documentation. At first, I didn’t actually understand how important docs are for software, especially for open source projects. However, during all this time I’ve learned how documentation has a huge impact on the open-source community and I’ve been thankful to have been able to do this. But, from the beginning, I asked myself “why am I the ‘right person’ for this?”
Improving Spyder’s documentation started as part of a NumFOCUS Small Development Grant awarded at the end of last year. The goal of the project was not only to update the documentation for Spyder 4, but also to make it more user-friendly, so users can understand Spyder’s key concepts and
(continued...)Keras Loss Functions: Everything You Need To Know
You’ve created a deep learning model in keras, you prepared the data and now you are wondering which loss you should choose for your problem. We’ll get to that in a second but first what is a loss function? In deep learning, the loss is computed to get the gradients with respect to model weights […]
The post Keras Loss Functions: Everything You Need To Know appeared first on neptune.ai.
The Best Sacred + Omniboard Alternatives
In this post, we’ll show you the best alternatives to Sacred + Omniboard. Since you came across this article, you may already know what these tools are made for. But let’s analyze them shortly as it could be helpful in better defining your needs and choosing the best alternative for you. It would be a […]
The post The Best Sacred + Omniboard Alternatives appeared first on neptune.ai.
Writing docs is not just writing docs
I joined the Spyder team almost two years ago, and I never thought I was going to end up working on docs. Six months ago I started a project with CAM Gerlach and Carlos Cordoba to improve Spyder’s documentation. At first, I didn’t actually understand how important docs are for software, especially for open source projects. However, during all this time I’ve learned how documentation has a huge impact on the open-source community and I’ve been thankful to have been able to do this. But, from the beginning, I asked myself “why am I the ‘right person’ for this?”
Read more… (3 min remaining to read)
This Week in Machine Learning: AI & Economy, ML Engineers, Criminal Justice, and More
Algorithms and numbers are all around us, we just don’t realize how much it affects our daily lives or even actions. What has happened in the machine learning realm over the last week? If you’re interested in how machine learning is changing our world, make sure to check out our weekly roundup. News, interesting stories. Read, […]
The post This Week in Machine Learning: AI & Economy, ML Engineers, Criminal Justice, and More appeared first on neptune.ai.
Interview with a Lead Data Scientist: Gabriel Preda
In the latest episode of our podcast, Machine Learning that Works, I had a great pleasure to talk to Gabriel Preda, a Lead Data Scientist at Endava and a Kaggle Grandmaster. We talked about: His work in Endava, NLP, Kaggle competitions, And how to put yourself in a position to be always learning. For those […]
The post Interview with a Lead Data Scientist: Gabriel Preda appeared first on neptune.ai.
The Best Pachyderm Alternatives
Pachyderm is a data science platform that helps to control an end-to-end machine learning life cycle. It comes in three different versions, Community Edition (open-source, with ability to be deployed anywhere), Enterprise Edition (complete version-controlled platform), and Hub Edition (a hosted version, still in beta). Here’s what you can do with Pachyderm in a nutshell: […]
The post The Best Pachyderm Alternatives appeared first on neptune.ai.
How to Track Hyperparameters of Machine Learning Models?
Machine learning algorithms are tunable by multiple gauges called hyperparameters. Recent deep learning models are tunable by tens of hyperparameters, that together with data augmentation parameters and training procedure parameters create quite complex space. In the reinforcement learning domain, you should also count environment params. Data scientists should control hyperparameter space well in order to […]
The post How to Track Hyperparameters of Machine Learning Models? appeared first on neptune.ai.
This Week in Machine Learning: AI & Ethics, Brain Tumors, and Wildlife Protection
What has happened in the machine learning world over the last week? If you’re interested in the latest trends, check out the weekly roundup. News, analysis, interesting stories. Read, learn, and get inspired! Here are the best picks from the last week from the world of machine learning. Enjoy the read! Weekly Roundup: June 17-29 […]
The post This Week in Machine Learning: AI & Ethics, Brain Tumors, and Wildlife Protection appeared first on neptune.ai.
Creating a Portable Python Environment from Imports
Python environments provide sandboxes in which packages can be added. Conda helps us deal with the requirements and dependencies of those packages. Occasionally we find ourselves working in a constrained remote machine which can make development challenging. Suppose we wanted to take our exact dev environment on the remote machine and recreate it on our local machine. While conda relieves the package dependency challenge, it can be hard to reproduce the exact same environment.
Read more… (3 min remaining to read)
Best Practices for Dealing with Concept Drift
You trained a machine learning model, validated its performance across several metrics which are looking good, you put it in production, and then something unforeseen happened (a pandemic like COVID-19 arrived) and the model predictions have gone crazy. Wondering what happened? You fell victim to a phenomenon called concept drift. But don’t feel bad as […]
The post Best Practices for Dealing with Concept Drift appeared first on neptune.ai.
Ibis: an idiomatic flavor of SQL for Python programmers
Ibis is a mature open-source project that has been in development for about 5 years; it currently has about 1350 stars on Github. It provides an interface to SQL for Python programmers and bridges the gap between remote storage & execution systems. These features provide authors the ability to:
Read more… (11 min remaining to read)
Sobre o livro “Uma História de Desigualdade”
Finalizei a leitura do premiado livro do Pedro de Souza, “Uma História de Desigualdade – A Concentração de Renda entre os Ricos no Brasil 1926 – 2013“, baseado na tese que defendeu no programa de sociologia da UnB. É um livro de fôlego e que faz jus a todos os elogios que recebeu desde o… Continue a ler »Sobre o livro “Uma História de Desigualdade”This Week in Machine Learning: Onboarding AI, Top Leaders, ML & Dogs, and More
Data science is great. It brings a lot of good to the world, helps in developing new ways of aiding people in their work, daily lives, and improves how things function. Each day something new and fascinating happens. As every week, we’ve gathered for you the best articles on machine learning. Check out the weekly […]
The post This Week in Machine Learning: Onboarding AI, Top Leaders, ML & Dogs, and More appeared first on neptune.ai.
How to Track Machine Learning Model Metrics in Your Projects
It is crucial to keep track of evaluation metrics for your machine learning models to: understand how your model is doing be able to compare it with previous baselines and ideas understand how far you are from the project goals “If you don’t measure it you can’t improve it.” But what should you keep track […]
The post How to Track Machine Learning Model Metrics in Your Projects appeared first on neptune.ai.
Thanking the people behind Spyder 4
This blogpost was originally published on the Quansight Labs website.
After more than three years in development and more than 5000 commits from 60 authors around the world, Spyder 4 finally saw the light on December 5, 2019! I decided to wait until now to write a blogpost about it because shortly after the initial release, we found several critical performance issues and some regressions with respect to Spyder 3, most of which are fixed now in version 4.1.3, released on May 8th 2020.
This new release comes with a lengthy list of user-requested features aimed at providing an enhanced development experience at the level of top general-purpose editors and IDEs, while strengthening Spyder's specialized focus on scientific programming in Python. The interested reader can take a look at some of them in previous blog posts, and in detail in our Changelog. However, this post is not meant to describe those improvements, but to acknowledge all people that contributed
(continued...)Technical discussions are hard; a few tips
Note
This post discuss the difficulties of communicating while developing open-source projects and tries to gives some simple advice.
A large software project is above all a social exercise in which technical experts try to reach good decisions together, for instance on github pull requests. But communication is difficult, in …
Tidynamics, what use?
In 2018 I published small Python library, tidynamics. The scope was deliberately limited: compute the typical correlation functions for stochastic and molecular dynamics: the autocorrelation and the mean-square displacement. Two years later, I wonder about its usage.
Moderna, IMC Renew NumFOCUS Corporate Sponsorships
Monday, May 18th, 2020 Two NumFOCUS corporate supporters recently made fresh commitments to our open source mission. Trading firm IMC and biotechnology company Moderna Therapeutics each renewed their corporate sponsorships earlier this month. Both companies have supported NumFOCUS since 2018 at our Silver and Bronze sponsorship levels, respectively. Asked about his company’s decision to partner […]
The post Moderna, IMC Renew NumFOCUS Corporate Sponsorships appeared first on NumFOCUS.
NumFOCUS Projects helping combat the COVID-19 pandemic
Open source tools are uniquely positioned to help combat the ongoing COVID-19 pandemic through their adaptable and collaborative nature. NumFOCUS sponsored and affiliated projects are being used on a global scale to meet the needs of researchers and data scientists. Our projects are being used in groundbreaking scientific efforts to create response models, visualize and […]
The post NumFOCUS Projects helping combat the COVID-19 pandemic appeared first on NumFOCUS.
Lazy River of Curious Content 0
This is the first post of what I'm calling a Lazy
River of Curious Content.
This is a way to review stuff that I've been doing, dealing with, or find
interesting during the week recently (This was
originally written two weeks ago, May 3rd, my shoddy internet connectivity kept
me from posting it.). I'm loosely following the format that Justin Sherrill
uses with great effect over at
https://dragonflydigest.com
Learn NixOS by turning a Raspberry Pi into a Wireless Router Friend of the show, Anthony Scopatz, tried NixOS for the first time and provides a detailed report:
"While I had read the NixOS pamphlets, and listened politely when the faithful came knocking on my door at inconvenient times, I had never walked the path of functional Linux enlightenment myself"
Reading through that made me file away a todo of writing up how I use propellor (and why). But those todo sometimes just pile up for a while...
An interview of one of my long time nerd-crushes, Rob Pike. The questions focus on the Go programming
(continued...)sourmash databases as zip files, in sourmash v3.3.0
The feature that I'm most excited about in sourmash 3.3.0 is the ability to directly use compressed SBT search databases.
Previously, if you wanted to search (say) 100,000 genomes from GenBank, you'd have to download a several GB .tar.gz file, and then uncompress it out to ~20 GB before searching it. The time and disk space requirements for this were major barriers for teaching and use.
In v3.3.0, Luiz Irber fixed this by, first, releasing the niffler Rust library with Pierre Marijon, to read and write compressed files; second, replacing our old khmer Bloom filter nodegraph with a Rust implementation (sourmash PR #799); and, third, adding direct zip file storage (sourmash #648).
So, as of the latest release, you can do the following:
# install sourmash v3.3.0
conda create -y -n sourmash-demo \
-c conda-forge -c bioconda sourmash=3.3.0
# activate environment
conda activate sourmash-demo
# download the 25k GTDB release89 guide database (~1.4 GB)
curl -L https://osf.io/5mb9k/download > gtdb-release89-k31.sbt.zip
# grabLaKademy 2019
Em novembro passado, colaboradores latinoamericanos do KDE desembarcaram em Salvador/Brasil para participarem de mais uma edição do LaKademy – o Latin American Akademy. Aquela foi a sétima edição do evento (ou oitava, se você contar o Akademy-BR como o primeiro LaKademy) e a segunda com Salvador como a cidade que hospedou o evento. Sem problemas… Continue a ler »LaKademy 2019Akademy 2019
Em setembro de 2019 a cidade italiana de Milão sediou o principal encontro mundial dos colaboradores do KDE – o Akademy, onde membros de diferentes áreas como tradutores, desenvolvedores, artistas, pessoal de promo e mais se reúnem por alguns dias para pensar e construir o futuro dos projetos e comunidade(s) do KDE Antes de chegar… Continue a ler »Akademy 2019Highlights of the Ibis 1.3 release
Ibis 1.3 was just released, after 8 months of development work, with 104 new commits from 16 unique contributors. What is new? In this blog post we will discuss some important features in this new version!
First, if you are new to the Ibis framework world, you can check this blog post I wrote last year, with some introductory information about it.
Some highlighted features of this new version are:
- Addition of a
PySparkbackend - Improvement of geospatial support
- Addition of
JSON,JSONBandUUIDdata types - Initial support for
Python 3.8added and support forPython 3.5dropped - Added new backends and geospatial methods to the documentation
- Renamed the
mapdbackend toomniscidb
Read more… (9 min remaining to read)
Yellowbrick Update – April 2020
Yellowbrick released Version 1.1 on February 25, 2020. If you haven’t yet upgraded simply type pip install yellowbrick -U or conda install -c districtdatalabs yellow-brick into your terminal/command prompt to get it. The major improvement in v1.1 is introducing quick methods or one-liners to generate your favorite ML plots more quickly with Yellowbrick. Dr. Rebecca […]
The post Yellowbrick Update – April 2020 appeared first on NumFOCUS.
2020 PyData Conferences Update [COVID-19]
We wanted to give an update to our community regarding the upcoming 2020 PyData conferences. We have been closely monitoring the situation and to help ensure the safety of our community given the threat of the COVID-19 virus, the following in-person events have been postponed to 2021: PyData Miami PyData Amsterdam PyData LA PyData London PyData […]
The post 2020 PyData Conferences Update [COVID-19] appeared first on NumFOCUS.
Scientific Software Developer- Contract Basis [SunPy Project]
Scientific Software Developer- Contract Basis NumFOCUS is seeking a Scientific Software Developer to support the SunPy project. SunPy is a Python-based open source scientific software package supporting solar physics data analysis. This is a 1 year contract. The successful applicant will work to improve SunPy’s functionality. There are four main tasks: Report on […]
The post Scientific Software Developer- Contract Basis [SunPy Project] appeared first on NumFOCUS.
Thanking the people behind Spyder 4
After more than three years in development and more than 5000 commits from 60 authors around the world, Spyder 4 finally saw the light on December 5, 2019! I decided to wait until now to write a blogpost about it because shortly after the initial release, we found several critical performance issues and some regressions with respect to Spyder 3, most of which are fixed now in version 4.1.2, released on April 3rd 2020.
Read more… (3 min remaining to read)
Creating the ultimate terminal experience in Spyder 4 with Spyder-Terminal
This blogpost was originally published on the Quansight Labs website.
The Spyder-Terminal project is revitalized! The new 0.3.0 version adds numerous features that improve the user experience, and enhances compatibility with the latest Spyder 4 release, in part thanks to the improvements made in the xterm.js project.
First, we were able to update all the old JavaScript files to use ES6/JSX syntax and the tests for the client terminal. This change simplified the code base and maintenance and allows us to easily extend the project to new functionalities that the xterm.js API offers. In order to compile this code and run it inside Spyder, we migrated our deployment to Webpack.
Multiple shells per operating systemIn the new release, you now have the ability to configure which shell to use in the terminal. On Linux and UNIX systems, bash, sh, ksh, zsh, csh, pwsh, tcsh, screen, tmux, dash and rbash are supported, while cmd and powershell are the
Introducing ndindex, a Python library for manipulating indices of ndarrays
One of the most important features of NumPy arrays is their indexing
semantics. By "indexing" I mean anything that happens inside square brackets,
for example, a[4::-1, 0, ..., [0, 1], np.newaxis]. NumPy's index semantics
are very expressive and powerful, and this is one of the reasons the library
is so popular.
Index objects can be represented and manipulated directly. For example, the
above index is (slice(4, None, -1), 0, Ellipsis, [0, 1], None). If you are
any author of a library that tries to replicate NumPy array semantics, you
will have to work with these objects. However, they are often difficult to
work with:
-
The different types that are valid indices for NumPy arrays do not have a uniform API. Most of the types are also standard Python types, such as
tuple,list,int, andNone, which are usually unrelated to indexing. -
Those objects that are specific to indexes, such as
sliceandEllipsisdo not make any assumptions about their underlying semantics. For example, Python lets you createslice(None, None, 0)orslice(0, 0.5)
Software and workflow development practices (April 2020 update)
Over the last 10-15 years, I've blogged periodically about how my lab develops research software and build scientific workflows. The last update talked a bit about how we've transitioned to snakemake and conda for automation, but I was spurred by an e-mail conversation into another update - because, y'all, it's going pretty well and I'm pretty happy!
Below, I talk through our current practice of building workflows and software. These procedures work pretty well for our (fairly small) lab of people who mostly work part-time on workflow and software development. By far the majority of our effort is usually spent trying to understand the results of our workflows; except in rare cases, I try to guide people to spend at most 20% of their time writing new analysis code - preferably less.
Nothing about these processes ensures that the scientific output is correct or useful, of course. While scientific correctness of computational workflows necessarily depends (often critically) on the correctness of the
(continued...)LaKademy 2019
Past November 2019 KDE fellows from Latin-America arrived in Salvador – Brazil to attend an one more edition of LaKademy – the Latin American Akademy. That was the 7th edition of the event (or the 8th, if you count Akademy-BR as the first LaKademy) and the second one with Salvador as host city. No problem… Continue a ler »LaKademy 2019PyTorch TensorIterator Internals
PyTorch is one of the leading frameworks for deep learning. Its core data
structure is Tensor, a multi-dimensional array implementation with many
advanced features like auto-differentiation. PyTorch is a massive
codebase (approx. a million lines of
C++, Python and CUDA code), and having a method for iterating over tensors in a
very efficient manner that is independent of data type, dimension, striding and
hardware is a critical feature that can lead to a very massive simplification
of the codebase and make distributed development much faster and smoother. The
TensorIterator
C++ class within PyTorch is a complex yet useful class that is used for
iterating over the elements of a tensor over any dimension and implicitly
parallelizing various operations in a device independent manner.
It does this through a C++ API that is independent of type and device of the
tensor, freeing the programmer of having to worry about the datatype or device
when writing iteration logic for PyTorch tensors. For those coming from the
NumPy universe, NpyIter is a close cousin of TensorIterator.
This
Is it getting better yet? An optimistic visual guide to the Coronavirus pandemic
As the apocalypse rumbles on, I found myself wondering "Is it getting any better?"
Daily updates of spiralling case numbers (and worse, deaths) does little to give a sense of whether we're getting to, or already past, the worst of it.
To answer that question for myself and you, I …
How to give a bad online talk
Today at lab meeting, I wanted to brainstorm about how to give good online talks, because I'm giving a few remote talks in the next month. Tracy suggested that perhaps I should demonstrate a bad talk first, just to get everyone on the same page.
So I did!
...enjoy? It's short, and not TOO painful if you show up with low expectations!
First, let me say that we were tremendously ...inspired by Greg Wilson's How to Teach Badly and How to Teach Badly (part 2)!
So here's what I did --
I put together a few slides on some stuff that I'd been working on recently, so it would look reasonable.
My initial screen opened with a private Twitter message up, to mimic inadvertent content sharing :).
I started out with "I didn't have a lot of time to prepare for this meeting so apologies for some of the slides."
My slide theme was very hard to read - bad fonts and colors.
A
(continued...)On the Link Between Polynomials and Optimization
There's a fascinating link between minimization of quadratic functions and polynomials. A link that goes deep and allows to phrase optimization problems in the language of polynomials and vice versa. Using this connection, we can tap into centuries of research in the theory of polynomials and shed new light on …
pheriday 3: infrastructure
img { display: block; margin-left: auto; margin-right: auto; } .thumb-link:hover { box-shadow: 0px 0px 8px #000; } .thumbnail { width: 80px; height: auto; margin: 0; padding: 0; display: block; } .thumb-link { display: inline-block; border: 5px solid #FFF; } .thumb-link:hover { box-shadow: 0px 0px 8px #000; } .thumbnail { width: 80px; height: auto; margin: 0; padding: 0; display: block; } .lightbox { opacity: 0; position: fixed; z-index: -1; width: 100%; height: 100%; text-align: center; top: 0; left: 0; background: rgba(0, 0, 0, 0.8); -webkit-transition: opacity 0.0s ease-out 0s; transition: opacity 0.0s ease-out 0s; } .lightbox img { (continued...)Documentation as a way to build Community
As a long time user and participant in open source communities, I've always known that documentation is far from being a solved problem. At least, that's the impression we get from many developers: "writing docs is boring"; "it's a chore, nobody likes to do it". I have come to realize I'm one of those rare people who likes to write both code and documentation.
Nobody will argue against documentation. It is clear that for an open-source software project, documentation is the public face of the project. The docs influence how people interact with the software and with the community. It sets the tone about inclusiveness, how people communicate and what users and contributors can do. Looking at the results of a “NumPy Tutorial” search on any search engine also gives an idea of the demand for this kind of content - it is possible to find documentation about how to read the NumPy documentation!
I've started working at Quansight in January,
PyData COVID-19 Response
The safety and well-being of our community are extremely important to us. We have therefore decided to postpone all PyData conferences scheduled to take place until the end of June: PyData Miami PyData London PyData Amsterdam We have been closely monitoring the situation and believe this is the best action to take based on the […]
The post PyData COVID-19 Response appeared first on NumFOCUS.
uarray: GSoC Participation
I'm pleased to announce that uarray is participating in GSoC '20 as a sub-organization under the umbrella of the Python Software Foundation. Our ideas page is up here, go take a look and see if you (or someone you know) is interested in participating, either as a student or as a mentor.
Prasun Anand and Peter Bell and myself will be mentoring, and we plan to take a maximum of two students, unless more community mentors show up.
There have been quite a few pull requests already to qualify from prospective students, some even going as far as to begin the work described in the idea they plan to work on.
We're quite excited by the number of students who have shown an interest in participating, and we look forward to seeing excellent applications! What's more exciting, though, are some of the first contributions from people not currently at Quansight, in the true spirit of open-source software!
Planned architectural work for PyData/Sparse
A lot of behind the scenes work has been taking place on PyData/Sparse. Not so much in terms of code, more in terms of research and community/team building. I've more-or-less decided to use the structure and the research behind the Tensor Algebra Compiler, the work of Fredrik Kjolstad and his collaborators at MIT. 🙇🏻♂️ To this end, I've read/watched the following talks and papers:
Read more… (2 min remaining to read)
Statement on Coronavirus
As you are aware, the Coronavirus (COVID-19) is a topic of frequent and ongoing discussions. We would like to provide an update on our status and policies as well as provide resources for additional information. As of today, our event schedule remains as posted on event sites. Any changes or updates will be immediately shared. […]
The post Statement on Coronavirus appeared first on NumFOCUS.
Some snakemake hacks for dealing with large collections of files
This winter quarter I taught my usual graduate-level introductory bioinformatics lab at UC Davis, GGG 201(b), for the fourth time. The course lectures are given by Megan Dennis and Fereydoun Hormozdiari, and I do a largely separate lab that aims to teach the basics of practical variant calling, de novo assembly, and RNAseq differential expression.
I also co-developed and co-taught a new course, GGG 298 / Tools for Data Intensive Research, with Shannon Joslin, a graduate student here in Genetics & Genomics who (among other things) took GGG 201(b) the first time I offered it. GGG 298 is a series of ten half-day workshops where we teach shell, conda, snakemake, git, RMarkdown, etc - you can see the syllabus for GGG 298 here.
This time around, I did a complete redesign of the GGG 201(b) lab (see syllabus) to focus on using snakemake workflows.
I'm 80% happy with how it went - there's some overall fine tuning to be done, and snakemake has some corners that need more explaining than other corners, but I think the
(continued...)Akademy 2019
Past September the Italian city of Milan hosted the KDE contributors meeting called Akademy, the main KDE conference where contributors from different areas like translators, developers, artists, promoters and more stay together for some days thinking and building the future of KDE projects and community(ies). Firstly before Akademy I departed from Brazil to Portugal to… Continue a ler »Akademy 2019Announcing JupyterCon 2020
NumFOCUS is excited to be a part of JupyterCon 2020. JupyterCon will be held August 10 – 14 in Berlin, Germany at the Berlin Conference Center. We invite you to participate in this exciting community event! Read the full announcement here. JupyterCon 2020 is an event brought to you in partnership by Project Jupyter and NumFOCUS.
The post Announcing JupyterCon 2020 appeared first on NumFOCUS.
MDAnalysis joins NumFOCUS Sponsored Projects
NumFOCUS is pleased to announce the newest addition to our fiscally sponsored projects: MDAnalysis MDAnalysis is a Python library for the analysis of computer simulations of many-body systems at the molecular scale, spanning use cases from interactions of drugs with proteins to novel materials. It is widely used in the scientific community and is written […]
The post MDAnalysis joins NumFOCUS Sponsored Projects appeared first on NumFOCUS.
Two talks at JGI in May: sourmash, spacegraphcats, and disease associations in the human microbiome.
Hello all! I'm giving two metagenomics talks - a tech talk and a bio talk - at the Joint Genome Institute on May 7, 2020. The abstracts are below.
The JGI just moved to a new building at LBNL, so these talks are much more accessible to the UC Berkeley and LBNL communities than they would have been a year ago. I hope interested people can make it!
The talks will be in the afternoon on May 7th at the Integrative Genomics Building, LBNL Bldg 91-310. I've put the tentative times down. I'll update this post with final times and contact information for security + parking passes closer to the day.
Bio talk: Novel approaches to metagenome analysis reveal microbial signatures of IBD(This will be the Science and Technology seminar, 3-4pm on May 7.)
Inflammatory bowel disease (IBD) is a spectrum of diseases characterized by chronic inflammation of the intestines; it is likely caused by host-mediated inflammatory responses at least in part elicited by microorganisms. As of 2015, 1.3% of US adults have been diagnosed
(continued...)Advancing research software in the UK through an SSI fellowship
I have selected as part of the 2020 cohort of Fellows of the Software Sustainability Institute!
The Institute cultivates world-class research with software. It's based at the universities of Edinburgh, Manchester, Southampton, and Oxford in the UK. Their motto says it all:
The SSI has a yearly fellowship program to fund the organization of communities around scientific software (creating of local user groups, workshops, hackathons, etc). Even more importantly, they organize several events to get current and past fellows in the same place doing awesome stuff. I'm really looking forward to this year's Collaborations Workshop (registration is open to all, not just fellows). I applied at the end of last year and was selected to join the 2020 cohort of fellows along with some truly amazing people.
My plan for the fellowship is to run
(continued...)Don't fund Software that doesn't exist
sourmash-oddify: a workflow for exploring contamination in metagenome-assembled genomes
(Thanks to Erich Schwarz, Taylor Reiter, and Donovan Parks for brainstorming and feedback on this stuff. Thanks also to Luiz Irber and Phillip Brooks for their work on sourmash!)
Yesterday, I posted about using k-mers and taxonomy to investigate Genbank genomes for potential contamination.
The underlying idea is pretty simple: look for subsets of k-mers that don't match the inferred taxonomy of the genome bins they're from, then analyze.
What started me down this path over two years ago (!!) was the use of the same underlying Tara Oceans metagenomic data for two separate papers, Tully et al., 2018 and Delmont et al., 2018. Both groups released their data early along with bioRxiv preprints, and it proved to be a treasure trove for my bioinformatics methods development - all of the sourmash lca functionality as well a lot of other functionality came from a series of about 14 blog posts examining these genomes.
I last left off with the
(continued...)Finding problematic bacterial/archaeal genomes using k-mers and taxonomy
(Happy New Year, everyone! Thanks on this blog post go out to Erich Schwarz and Taylor Reiter, for offering helpful suggestions and asking tough questions as I meandered through this work!)
Yesterday,
I posted
about using sourmash lca classify to taxonomically classify
bacterial and archaeal genomes quickly, and compared the results to
the full GTDB taxonomy. The tl;dr was that sourmash works pretty well
and returns results consistent with GTDB and GTDB-Tk, but that it
often doesn't classify as precisely as GTDB-Tk.
I was kind of expecting that at the species level, because there is a limit to the kind of precision that downsampled k-mers can achieve: the last 1-0.1% of nucleotide similarity can be a bit wobbly with sourmash (<- technical term).
But I was surprised to see the phylum and superkingdom level
limits. sourmash lca classify couldn't classify 235 genomes beyond
phylum level! What could be causing this?
I took a closer look at
GCF_001477405,
a genome tagged as Staphylococcus sciuri in Genbank. Using sourmash
lca summarize,
How does sourmash's lca classification routine compare with GTDB classifications?
Yesterday I posted about the GTDB taxonomy; we are now providing prepared databases that can be used with sourmash's taxonomy classification routines to classify genomes with GTDB.
The databases we posted are built from the dereplicated 25k GTDB genomes distributed as part of the GTDB-Tk classification toolkit, and not the full 145k classifications in GTDB. So they are smaller than they could be, and also potentially lower resolution. Moreover, sourmash uses k-mers instead of amino acids, which may lead to different classifications.
A good first question is, how well do classifications with sourmash
lca classify & 25k genomes compare to the full 145k classifications
in GTDB? This is basically a measure of generalizability - how
reliably can we infer the classifications of the 145k genomes from the
25k?
sourmash lca classify on Genbank to GTDB
I classified all 420k Genbank genomes using sourmash lca classify
with k=31, and I then wrote a script to compare the output to the GTDB
taxonomy. This involved some rather nasty identifier conversion which
sometimes failed, but we ended up with a good
Sourmash LCA databases now available for the GTDB taxonomy
I am happy to announce that we have made available prepared sourmash taxonomy ("LCA") databases for release 89 of the GTDB taxonomy.
The databases are available for download from the Open Science Framework in this project. There are prepared databases avaialble for k=21, k=31, and k=51.
What is the GTDB taxonomy?GTDB is a revised bacterial and archaeal taxonomy based on phylogenetic relations between proteins from approximately 25k genomes. You can read more about it here.
GTDB is an alternative to the NCBI taxonomy. It is used by (among others) MGnify, the EBI metagenomics resource.
What is sourmash?Sourmash is a research platform and bioinformatics tool for searching and analyzing genomes, based on a MinHash-inspired approach that allows genome similarity searches, genome containment searches, and compositional analysis of k-mers in large sequence data sets. You can read more about it here.
What do these databases let you do?There are three immediate uses for these databases:
-
you can use the
sourmash lca classifyroutine (and other LCA commands) to do taxonomic classification of genomes
Two PhD studentships at the University of Liverpool
I have two open positions for funded studentships at the University of Liverpool. Applications are open until 10 January 2020.
Project descriptionsFollow the links for more detailed versions.
Bringing machine learning techniques to geophysical data processing
The goal of this project is to investigate the use of existing machine learning techniques to process gravity and magnetics data using the Equivalent Layer Method. The methods and software developed during this project can be applied to process large amounts of gravity and magnetics data, including airborne and satellite surveys, and produce data products that can enable further scientific investigations. Examples of such data products include global gravity gradient grids from GOCE satellite measurements, regional magnetic grids for the UK, gravity grids for the Moon and Mars, etc.
Large-scale mapping of the thickness of the
(continued...)SunPy Receives NASA Grant, Helps Generate Parker Solar Probe Results
The post SunPy Receives NASA Grant, Helps Generate Parker Solar Probe Results appeared first on NumFOCUS.
NumFOCUS Kicks Off Year-End Fundraising with $2,500 Gift
Giving Tuesday 2019 marked the start of NumFOCUS’s year-end fundraising campaign, and this year’s effort began with a major donation from the organization’s Board President, Andy Terrel. “Today I’m pledging a gift of $2,500 to help kick off the NumFOCUS end-of-year fundraising campaign,” Terrel wrote yesterday in an e-mail to the NumFOCUS community. “I believe […]
The post NumFOCUS Kicks Off Year-End Fundraising with $2,500 Gift appeared first on NumFOCUS.
Stepping Onto a New Path…
The post Stepping Onto a New Path… appeared first on NumFOCUS.
Getting a big scientific prize for open-source software
Note
An important acknowledgement for a different view of doing science: open, collaborative, and more than a proof of concept.
A few days ago, Loïc Estève, Alexandre Gramfort, Olivier Grisel, Bertrand Thirion, and myself received the “Académie des Sciences Inria prize for transfer”, for our contributions to the scikit-learn project …
Variable Explorer improvements in Spyder 4
This blogpost was originally published on the Quansight Labs website.
Spyder 4 will be released very soon with lots of interesting new features that you'll want to check out, reflecting years of effort by the team to improve the user experience. In this post, we will be talking about the improvements made to the Variable Explorer.
These include the brand new Object Explorer for inspecting arbitrary Python variables, full support for MultiIndex dataframes with multiple dimensions, and the ability to filter and search for variables by name and type, and much more.
It is important to mention that several of the above improvements were made possible through integrating the work of two other projects. Code from gtabview was used to implement the multi-dimensional Pandas indexes, while objbrowser was the foundation of the new Object Explorer.
New viewer for arbitrary Python objectsFor Spyder 4 we added a long-requested feature: full support for inspecting any kind of Python object through the Variable
(continued...)File management improvements in Spyder 4
This blogpost was originally published on the Quansight Labs website.
Version 4.0 of Spyder is almost ready! It has been in the making for well over two years, and it contains lots of interesting new features. We will focus on the Files pane in this post, where we've made several improvements to the interface and file management tools.
Simplified interfaceIn order to simplify the Files pane's interface, the columns corresponding to size and kind are hidden by default. To change which columns are shown, use the top-right pane menu or right-click the header directly.
First, we added the ability to associate different external applications with specific file extensions they can open. Under the File associations tab of the Files preferences pane, you can add file types and set the external program used to open each of them by default.
Once you've set this up, files will automatically launch in the associated application when opened from the Files pane in Spyder.
(continued...)This Week in Glean: November 1, 2019
When data goes wrong
Loan Amortisation Schedule using R and Python
When you take a loan from bank at x% annual interest rate for N number of years. Bank calculates monthly (or quarterly) instalments based on the following factors :
- Loan Amount
- Annual Interest Rate
- Number of payments per year
- Number of years for loan to be repaid in instalments
Introducing napari: a fast n-dimensional image viewer in Python
I'm really excited to finally, officially, share a new(ish) project called napari with the world. We have been developing napari in the open from the very first commit, but we didn't want to make any premature fanfare about it… Until now. It's still alpha software, but for months now, both the core napari team and a few collaborators/early adopters have been using napari in our daily work. I've found it life-changing.
The backgroundI've been looking for a great nD volume viewer in Python for the better part of
a decade. In 2009, I joined Mitya Chklovskii's lab and the FlyEM team at the
Janelia [Farm] Research Campus to work on the segmentation of 3D electron
microscopy (EM) volumes. I started out in Matlab, but moved to Python pretty
quickly and it was a very smooth transition (highly recommended! ;). Looking at
my data was always annoying though. I was either looking at single 2D slices
using matplotlib.pyplot.imshow, or saving the volumes in VTK format and
loading them into ITK-SNAP — which worked ok
O SERPRO e a validação de documentos digitais
No rascunho do post anterior sobre os documentos digitais no Brasil acabei escrevendo bastante sobre o papel do SERPRO nesse processo – tanto que decidi separá-lo em um post próprio. Com o lançamento do e-Título foi necessário para o TSE criar uma maneira de validar o documento digital para evitar fraudes. A tecnologia adotada foi… Continue a ler »O SERPRO e a validação de documentos digitaisOs documentos digitais (no plural) do Brasil
Já faz algum tempo o Brasil está passando por um processo de digitalização dos documentos oficiais utilizados por pessoas físicas. Entretanto, o que antes se anunciava como uma possível convergência dos mais diferentes documentos para um documento único, que serviria para tudo, passou a acontecer a digitalização de cada documento específico através do desenvolvimento de… Continue a ler »Os documentos digitais (no plural) do BrasilHow to Evaluate the Logistic Loss and not NaN trying
A naive implementation of the logistic regression loss can results in numerical indeterminacy even for moderate values. This post takes a closer look into the source of these instabilities and discusses more robust Python implementations.
hljs.initHighlightingOnLoad(); MathJax.Hub.Config({ extensions: ["tex2jax.js"], jax: ["input/TeX", "output/HTML-CSS"], tex2jax: { inlineMath …Uvas Gold 200
My poem about a rainy 200k was published in the Fall 2019 issue of American Randonneur (a quarterly magazine published by Randonneurs USA)
I've been doing samizdat poetry for as long as I've had a web presence (since 1999), but I am now officially a published poet! (I am deliberately not counting the embarrasing hackjob that was published in a youth anthology when I was in 8th grade.)
You can find "Uvas Gold 200" on page 26 - either directly on this skeuomorphic leafing viewer or the PDF, but I'm republishing both the exposition blurb and the poem below. If you prefer to listen, I recorded a reading of it that you can download in different flavors: a local audio only, a local video, or the embeded video version below.
(continued...)Uvas Gold 200k starts and ends in Fremont, CA and was held on Saturday, December 1st, 2018. The ride frontloads the climbing by going nearly half-way up Mount
Grupo de Estudos do Laboratório Amazônico de Estudos Sociotécnicos – UFPA
Eu e o prof. Leonardo Cruz da Faculdade de Ciências Sociais estamos juntos trabalhando no desenvolvimento do Laboratório Amazônico de Estudos Sociotécnicos da UFPA. Nossa proposta é realizar leituras e debates críticos sobre o tema da sociologia da tecnologia, produzir pesquisas teóricas e empíricas na região amazônica sobre as relações entre tecnologia e sociedade, e… Continue a ler »Grupo de Estudos do Laboratório Amazônico de Estudos Sociotécnicos – UFPApelas ruas de Belém
as vezes pelas ruas de Belém não sei se sou eu ou minha mãe quem está aliSpyder 4.0: Kite integration is here
This blogpost was originally published on the Quansight Labs website.
Note: Kite is sponsoring the work discussed in this blog post, and in addition supports Spyder 4.0 development through a Quansight Labs Community Work Order.
As part of our next release, we are proud to announce an additional completion client for Spyder, Kite. Kite is a novel completion client that uses Machine Learning techniques to find and predict the best autocompletion for a given text. Additionally, it collects improved documentation for compiled packages, e.g. Matplotlib, NumPy and SciPy, that cannot be obtained easily by using traditional code analysis packages such as Jedi. Although Kite is not open source like Spyder, you can download it without charge at the Kite website.
By incorporating Kite into Spyder, we will improve and provide the ultimate autocompletion and signature retrieval experience for most of the scientific Python stack and beyond. For instance, let’s take a look at the following PyTorch completion. While
(continued...)An initial report on the Common Fund Data Ecosystem
For the past 6 months or so, I've been working with a team of people on a project called the Common Fund Data Ecosystem. This is a targeted effort within the NIH Common Fund (CF) to improve the Findability, Accessibility, Interoperability, and Reusability - a.k.a. "FAIRness" - of the data sets hosted by their Data Coordinating Centers.
(You can see Dr. Vivien Bonazzi's presentation if you're interested in more details on the background motivation of this project.)
I'm thrilled to announce that our first report is now available! This is the product of a tremendous data gathering effort (by many people), four interviews, and an ensuing distillation and writing effort with Owen White and Amanda Charbonneau. To quote,
(continued...)This assessment was generated from a combination of systematic review of online materials, in-person site visits to the Genotype Tissue Expression (GTEx) DCC and Kids First, and online interviews with Library of Integrated Network-Based Cellular Signatures (LINCS) and Human Microbiome Project
Object Oriented Programming in Python : Learn by Examples
In the above example, car is an object. Functions are called methods in OOP world. Characteristics are attributes (properties). Technically attributes are variables or values related to the state of the object whereas methods
Precision Recall Curve Simplified
Comparing two genome binnings quickly with sourmash
tl;dr? Compare and cluster two collections of 1000+ metagenome-assembled genomes in a few minutes with sourmash!
A week ago, someone e-mailed me with an interesting question: how can we compare two collections of genome bins with sourmash?
Why would you want to do this? Well, there's lots of reasons! The main one that caught my attention is comparing genomes extracted from metagenomes via two different binning procedures - that's where I started almost two years ago, with two sets of bins extracted from the Tara ocean data. You might also want to merge bins that were similar to produce a (hopefully) more complete bin, or you could intersect bins that were similar to produce a consensus bin that might be higher quality, or you could identify bins that were in one collection and not in the other, to round out your collection.
I'm assuming this is done by lots of workflows - I note, for example, that the metaWRAP workflow includes a 'bin refinement' step that must do something like
(continued...)Calculate KS Statistic with Python
In data science, it compares the cumulative distribution of events and non-events and KS is where there is a maximum difference between the two distributions. In simple words, it helps us to understand how well our predictive model is able to discriminate between events and
A Complete Guide to Python DateTime Functions
- datetime
- date
- time
- timedelta
People who have no experience of working with real-world datasets might have not encountered date columns. They might be under impression that working with dates is rarely used and not so
What are *args and **kwargs and How to use them
args is a short form of arguments. With the use of *args python takes any number of arguments in user-defined function and converts user inputs to a tuple named args. In other words, *args means zero or more arguments which are stored in a tuple named args. When you define function without *args, it has a fixed number of inputs which means it cannot accept more (or less) arguments than you defined in the function.
In the example code below, we are creating a very basic function which adds two numbers. At the same time, we created a
Python : 10 Ways to Filter Pandas DataFrame
- Select all the active customers whose accounts were opened
Python Dictionary Comprehension with Examples
- Keys : Think about columns in tables. It must be unique (like column names cannot be duplicate)
- Values : It is similar to rows in tables. It can be duplicate.
To extract keys, values and structure of dictionary, you can submit the following commands.
d = {'a': [1,2], 'b': [3,4], 'c': [5,6]}
d.keys() # 'a', 'b', 'c'
d.values() # [1, 2], [3, 4], [5,
HTML outputs in Jupyter
SummaryUser interaction in data science projects can be improved by adding a small amount of visual deisgn.
To motivate effort around visual design we show several simple-yet-useful examples. The code behind these examples is small and accessible to most Python developers, even if they don’t have much HTML experience.
This post in particular focuses on Jupyter’s ability to add HTML output to any object. This can either be full-fledged interactive widgets, or just rich static outputs like tables or diagrams. We hope that by showing examples here we will inspire some throughts in other projects.
This post was supported by replies to this tweet. The rest of this post is just examples.
IrisI originally decided to write this post after reading another blogpost from the UK Met office, where they included the HTML output of their library Iris in a a blogpost
(work by Peter Killick, post by Theo McCaie)
The fact that the output provided by an interactive session is the same output that you would provide in a published result helps everyone. The interactive
(continued...)Python list comprehension with Examples
f(x) = x2, f(x) will always return the same result with the same x value. The function has no "side-effect" which means an operation has no effect on a variable/object that is outside the intended usage. "Side-effect" refers to leaks in your code which can modify a mutable data structure or variable. Functional programming is also good for parallel computing as there is no
Don't cite the No Free Lunch Theorem
Tldr; You probably shouldn’t be citing the "No Free Lunch" Theorem by Wolpert. If you’ve cited it somewhere, you might have used it to support the wrong conclusion. What it actually (vaguely) says is “You can’t learn from data without making assumptions”.The paper on the “No Free Lunch Theorem”, actually called "The Lack of A Priori Distinctions Between Learning Algorithms" is one of these papers that are often cited and rarely read, and I hear many people in the ML community refer to it when supporting the claim that “one model can’t be the best at everything” or “one model won’t always be better than another model”. The point of this post is to convince you that this is not what the paper or theorem says (at least not the one usually cited by Wolpert), and you should not cite this theorem in this context; and also that common versions cited of the "No Free Lunch" Theorem (continued...)
15 ways to read CSV file with pandas
pip install pandas. Next step is to load the package by running the following command. pd is an alias of pandas package. We will use it instead of full name "pandas". import pandas as pd
Matplotlib Tutorial : Learn with Examples in 3 hours
MATLAB programming language and provides a similar MATLAB like interface for graphics. The beauty of this library is that it integrates well with pandas package which is used for data manipulation. With the combination of these two libraries, you can easily perform data wrangling along with visualization and get valuable insights out of data. Like ggplot2 library in R, matplotlib library is the grammar of graphics in Python and most used library for charts in Python. 
Write Short Blogposts
I encourage my colleagues to write blogposts more frequently. This is for a few reasons:
-
It informs your broader community what you’re up to, and allows that community to communicate back to you quickly.
You communicating to the community fosters a sense of collaboration, openness, and trust. You gain collaborators, build momentum behind your work, and curate a body of knowledge that early adopters can consume to become experts quickly.
Getting feedback from your community helps you to course-correct early in your work, and stops you from wasting time in inefficient courses of action.
You can only work for a long time without communicating if you are either entirely confident in what you’re doing, or reckless, or both.
-
It increases your visibility, and so is good for your career.
I have a great job. I find my work to be both
How to encourage participation in teleconferences
(and/or how to run effective teleconferences!)
I participate in a lot of teleconferences, and some of them aren't very participatory, for various reasons. Recently a good friend asked for suggestions on how to open up the phone calls, and I came up with the below ideas. What am I missing? What did I get wrong?
First, post a meeting agenda with a medium amount of detail, well in advance ( > 24 hours).
- Posting an agenda in advance gives people time to think about things, if they are interested.
- The medium amount of detail (up to a paragraph) lets people understand what it’s about, see what the major issues/questions are, and think of questions or comments they may have.
- If the agenda is posted > 24 hours in advance, you can reasonably expect people to have read it, and if people want to add things to the agenda on the call you punt them to the next call instead.
Basically, if you spring a skeleton
(continued...)How to drop one or multiple columns from Pandas Dataframe
- Drop or Keep rows and columns
- Aggregate data by one or more columns
- Sort or reorder data
- Merge or append multiple dataframes
- String Functions to handle text data
- DateTime Functions to handle date or time format columns
import command. import pandas as pd
import numpy as np
df = pd.DataFrame(np.random.randn(6, 4), columns=list('ABCD'))
A B C D
0 -1.236438 -1.656038
String Functions in Python with Examples
The cost of an open source contribution
Open source is massively successful. Some say it’s eating the world, although to my ears that phrasing doesn’t sound entirely like a good thing. Open source maintainers are always in need of help, and over the past years I’ve seen a lot of focus on ways open source projects can grow their communities and gain new contributors. Guidance on how to go about finding new contributors is easily found. E.TDK-Micronas partners with Quansight to sponsor Spyder
This blogpost was originally published on the Quansight Labs website
TDK-Micronas is sponsoring Spyder development efforts through Quansight Labs. This will enable the development of some features that have been requested by our users, as well as new features that will help TDK develop custom Spyder plugins in order to complement their Automatic Test Equipment (ATE’s) in the development of their Application Specific Integrated Circuits (ASIC’s).
At this point it may be useful to clarify the relationship the role of Quansight Labs in Spyder's development and the relationship with TDK. To quote Ralf Gommers (director of Quansight Labs):
(continued...)"We're an R&D lab for open source development of core technologies around data science and scientific computing in Python. And focused on growing communities around those technologies. That's how I see it for Spyder as well: Quansight Labs enables developers to be employed to work on Spyder, and helps with connecting them to developers of other projects in similar situations. Labs should be an enabler to let the Spyder project, its community and individual developers grow.
Why citations are not enough for open source software
A few weeks ago I wrote about why you should cite open source tools. Although I think citations important, though, there are major problems in relying on them alone to support open source work.
The biggest problem is that papers describing a software library can only give credit to the contributors at the time that the paper was written. The preferred citation for the SciPy library is “Eric Jones, Travis Oliphant, Pearu Peterson, et al”, 2001. The “et al” is not an abbreviation here, but a fixed shorthand for all other contributors. Needless to say many, many people have contributed to the SciPy library since 2001 (GitHub counts 716 contributors as of this writing), and they are unable to get credit within the academic system for those contributions. (As an aside, Google counts about 1,200 citations to SciPy, which is a breathtaking undercounting of its value and influence, and reinforces my earlier point: cite open source software! Definitely don't use this post as an excuse not to cite it!!!)
Not surprisingly, we have had
Spyder 4.0 takes a big step closer with the release of Beta 2!
This blogpost was originally published on the Quansight Labs website
It has been almost two months since I joined Quansight in April, to start working on Spyder maintenance and development. So far, it has been a very exciting and rewarding journey under the guidance of long time Spyder maintainer Carlos Córdoba. This is the first of a series of blog posts we will be writing to showcase updates on the development of Spyder, new planned features and news on the road to Spyder 4.0 and beyond.
First off, I would like to give a warm welcome to Edgar Margffoy, who recently joined Quansight and will be working with the Spyder team to take its development even further. Edgar has been a core Spyder developer for more than two years now, and we are very excited to have his (almost) full-time commitment to the project.
Spyder 4.0 Beta 2 released!Since August 2018, when the first beta of the 4.x series was released, the Spyder development team has been
(continued...)The Role of a Maintainer
What are the expectations and best practices for maintainers of open source software libraries? How can we do this better?
This post frames the discussion and then follows with best practices based on my personal experience and opinions. I make no claim that these are correct.
Let us Assume External ResponsibilityFirst, the most common answer to this question is the following:
- Q: What are expectations on OSS maintainers?
- A: Nothing at all. They’re volunteers.
However, let’s assume for a moment that these maintainers are paid to maintain the project some modest amount, like 10 hours a week.
How can they best spend this time?
What is a Maintainer?Next, let’s disambiguate the role of developer, reviewer, and maintainer
-
Developers fix bugs and create features. They write code and docs and generally are agents of change in a software project. There are often many more developers than reviewers or maintainers.
-
Reviewers are known
Using GitHub for janky project reporting - some code
For the NIH Data Commons, we needed a way for 10 distinct teams to do reporting at the level of about 50-100 milestones per team, on a monthly basis.
Each team was already using different project management software internally, and we didn't want to require them to switch to something new. We also didn't need a lot of innate functionality in the project reporting system - basically, for each milestone we needed two statuses, "started" and "finished".
So we decided to go with something lightweight and simple that would support programmatic update and automated reporting: GitHub!
We chose to use GitHub for project reporting for several reasons. We were already using GitHub for content stuff, and everyone had accounts. We were also using GitHub for authentication control on static Web sites via a Heroku app.
So what we did was use the PyGithub package to write a script to take the project milestones (which were all in a spreadsheet) and load them
(continued...)My first DNF (Ft Bragg 600k)
It's been six years since my first ride with The San Francisco Randonneurs and four years since my first 200k. I've ridden 18 rides that are at least that distance since then (3x 300k, 2x 400k, 1x 600k), completing my first Super Randonneur Series (2-, 3-, 4-, and 600k in one year) last year after not riding much the year before that. And this weekend I had my first DNF result on the Fort Bragg 600k. I Did Not Finish.
The best response to my choice of abandoning the ride to enjoy the campground came from Peter Curley, who said "That was a very mature decision." A clear departure from typical randonneuring stubbornness and refusal to give up, I celebrated my decision to quit as a victory when I arrived at the campground and made my announcement to the volunteers. I think I was so energetic about it that they did not believe me. I was being kind to myself, to my body, and at peace with the decision by
(continued...)Should I Resign from My Full Professor Job to Work Fulltime on Cocalc?
PyCon2019 poem
I'm back in Cleveland for another Pycon. Yesterday was my first full day here. Along with Matt Seale, I was a helper at Matthias Bussonnier tutorial ("IPython and Jupyter in Depth: High productivity, interactive Python). The sticky system is efficient at signaling when someone in a classroom needs help, and a lot of folks don't know that this practice was popularized by Software Carpentry workshops and continues to be used at The Carpentries.
I stepped out for a coffee refill and bumped into a large contingent of Bloomberg folks I'd never met (Princeton office). I guess we have something like 90 people at the conference this year, and I made the usual and true remark about how I go to conferences to meet the other people who work at our company. Then after his tutorial concluded, Matthias and I bumped into Tracy Teal, exchanged some stickers, and chatted about The Carpentries, Jupyter, organizing conferences, governance and sponsorship models, and a bunch of other stuff.
Matthias was a
(continued...)Why you should cite open source tools
Every now and then, a moment or a sentence in a conversation sticks out at you, and lodges itself in the back of your brain for months or even years. In this case, the sentence is a tweet, and I fear that the only way to dislodge it is to talk about it publicly.
Last year, I complained on Twitter that a very prominent paper that was getting lots of attention used scikit-image, but failed to cite our paper. (Or the papers corresponding to many other open source packages.) I continued that scientists developing open source software depend on these citations to continue their work. (More on this in another post...) One response was that surely the developers of the open source scientific Python stack were not scientists per se, and that citations were not a priority for them.
I still sigh internally when I think of it.
That tweet manifests a pervasive perception that open source scientific software is written by God-like figures. These massively experienced software developers have easy access to funds
Python Lambda Function with Examples
anonymous function as the function can be defined without its name. They are a part of functional programming style which focus on readability of code and avoids changing mutable data. 
lambda arguments: expressionLambda function can have more than one argument but expression cannot be more than 1. The expression is evaluated and returned. Example
addition = lambda x,y: x + yIn the above python code,
addition(2,3) returns 5
x,y are the arguments and x + y is the expression that gets evaluated and returned.