Planet SciPy
Labs update and May highlights
Time flies when you're having fun. Here is an update of some of the highlights of my second month at Quansight Labs.
The making of a black hole image & GitHub SponsorsBoth Travis and myself were invited by GitHub to attend GitHub Satellite in Berlin. The main reason was that Nat Friedman (GitHub CEO) decided to spend the first 20 minutes of his keynote to highlight the Event Horizon Telescope's black hole image and the open source software that made that imaging possible. This included the scientific Python very prominently - NumPy, Matplotlib, Python, Cython, SciPy, AstroPy and other projects were highlighted. At the same time, Nat introduced new GitHub features like "used by", a triaging role and new dependency graph features and illustrated how those worked for NumPy. These features will be very welcome news to maintainers of almost any project.
The single most visible feature introduced was GitHub Sponsors:
I really enjoyed meeting Devon Zuegel, Product Manager of the Open Source Economy Team at GitHub, in person after previously having had
String Functions in Python with Examples
Anaconda Recognized as a May 2019 Gartner Peer Insights Customers’ Choice for Data Science and Machine Learning Platforms
The Anaconda team is excited to announce that we have been recognized as a May 2019 Gartner Peer Insights Customers’ Choice for Data Science and Machine Learning Platforms. According to Gartner, “The Gartner Peer Insights…
The post Anaconda Recognized as a May 2019 Gartner Peer Insights Customers’ Choice for Data Science and Machine Learning Platforms appeared first on Anaconda.
TDK-Micronas partners with Quansight to sponsor Spyder
TDK-Micronas is sponsoring Spyder development efforts through Quansight Labs. This will enable the development of some features that have been requested by our users, as well as new features that will help TDK develop custom Spyder plugins in order to complement their Automatic Test Equipment (ATE’s) in the development of their Application Specific Integrated Circuits (ASIC’s).
At this point it may be useful to clarify the relationship the role of Quansight Labs in Spyder's development and the relationship with TDK. To quote Ralf Gommers (director of Quansight Labs):
"We're an R&D lab for open source development of core technologies around data science and scientific computing in Python. And focused on growing communities around those technologies. That's how I see it for Spyder as well: Quansight Labs enables developers to be employed to work on Spyder, and helps with connecting them to developers of other projects in similar situations. Labs should be an enabler to let the Spyder project, its community and individual developers grow. And Labs provides mechanisms to attract and coordinate funding. Of course
metadsl: A Framework for Domain Specific Languages in Python
metadsl: A Framework for Domain Specific Languages in Python¶Hello, my name is Saul Shanabrook and for the past year or so I have been at Quansight exploring the array computing ecosystem. This started with working on the xnd project, a set of low level primitives to help build cross platform NumPy-like APIs, and then started exploring Lenore Mullin's work on a mathematics of arrays. After spending quite a bit of time working on an integrated solution built on these concepts, I decided to step back to try to generalize and simplify the core concepts. The trickiest part was not actually compiling mathematical descriptions of array operations in Python, but figuring out how to make it useful to existing users. To do this, we need to meet users where they are at, which is with the APIs they are already familiar with, like numpy. The goal of metadsl is to make it easier to tackle parts
Community-driven open source and funded development
Quansight Labs is an experiment for us in a way. One of our main aims is to channel more resources into community-driven PyData projects, to keep them healthy and accelerate their development. And do so in a way that projects themselves stay in charge.
This post explains one method we're starting to use for this. I'm writing it to be transparent with projects, the wider community and potential funders about what we're starting to do. As well as to explicitly solicit feedback on this method.
Community work ordersIf you talk to someone about supporting an open source project, in particular a well-known one that they rely on (e.g. NumPy, Jupyter, Pandas), they're often willing to listen and help. What you quickly learn though is that they want to know in some detail what will be done with the funds provided. This is true not only for companies, but also for individuals. In addition, companies will likely want a written agreement and some form of reporting about the progress of the work. To meet this
Measuring API usage for popular numerical and scientific libraries
Developers of open source software often have a difficult time understanding how others utilize their libraries. Having better data of when and how functions are being used has many benefits. Some of these are:
- better API design
- determining whether or not a feature can be deprecated or removed.
- more instructive tutorials
- understanding the adoption of new features
We wrote a general tool
python-api-inspect
to analyze any function/attribute call within a given set of
namespaces in a repository. This work was heavily inspired by a blog
post on inspecting method usage with
Google BigQuery
for pandas,
NumPy, and
SciPy. The previously mentioned work used
regular expressions to search for method usage. The primary issue with
this approach is that it cannot handle import numpy.random as rand;
rand.random(...) unless additional regular expressions are
constructed for each case and will result in false
positives. Additionally,
BigQuery is not a free resource.
Thus, this approach is not general enough and does not scale well with
the number of libraries that we would like to inspect function and
attribute usage.
A more robust
Intake: Discovering and Exploring Data in a Graphical Interface
Motivation Do you have data that you’d like people to be able to explore on their own? Are you always passing around snippets of code to load specific data files? These are problems that people…
The post Intake: Discovering and Exploring Data in a Graphical Interface appeared first on Anaconda.
Spyder 4.0 takes a big step closer with the release of Beta 2!
It has been almost two months since I joined Quansight in April, to start working on Spyder maintenance and development. So far, it has been a very exciting and rewarding journey under the guidance of long time Spyder maintainer Carlos Córdoba. This is the first of a series of blog posts we will be writing to showcase updates on the development of Spyder, new planned features and news on the road to Spyder 4.0 and beyond.
First off, I would like to give a warm welcome to Edgar Margffoy, who recently joined Quansight and will be working with the Spyder team to take its development even further. Edgar has been a core Spyder developer for more than two years now, and we are very excited to have his (almost) full-time commitment to the project.
Spyder 4.0 Beta 2 released!Since August 2018, when the first beta of the 4.x series was released, the Spyder development team has been working hard on our next release. Over the past year, we've
Spyder 4.0 takes a big step closer with the release of Beta 2!
This blogpost was originally published on the Quansight Labs website
It has been almost two months since I joined Quansight in April, to start working on Spyder maintenance and development. So far, it has been a very exciting and rewarding journey under the guidance of long time Spyder maintainer Carlos Córdoba. This is the first of a series of blog posts we will be writing to showcase updates on the development of Spyder, new planned features and news on the road to Spyder 4.0 and beyond.
First off, I would like to give a warm welcome to Edgar Margffoy, who recently joined Quansight and will be working with the Spyder team to take its development even further. Edgar has been a core Spyder developer for more than two years now, and we are very excited to have his (almost) full-time commitment to the project.
Spyder 4.0 Beta 2 released!Since August 2018, when the first beta of the 4.x series was released, the Spyder development team has been
(continued...)The Role of a Maintainer
What are the expectations and best practices for maintainers of open source software libraries? How can we do this better?
This post frames the discussion and then follows with best practices based on my personal experience and opinions. I make no claim that these are correct.
Let us Assume External ResponsibilityFirst, the most common answer to this question is the following:
- Q: What are expectations on OSS maintainers?
- A: Nothing at all. They’re volunteers.
However, let’s assume for a moment that these maintainers are paid to maintain the project some modest amount, like 10 hours a week.
How can they best spend this time?
What is a Maintainer?Next, let’s disambiguate the role of developer, reviewer, and maintainer
-
Developers fix bugs and create features. They write code and docs and generally are agents of change in a software project. There are often many more developers than reviewers or maintainers.
-
Reviewers are known
Two Sigma Renews Corporate Sponsorship of NumFOCUS
The post Two Sigma Renews Corporate Sponsorship of NumFOCUS appeared first on NumFOCUS.
Using GitHub for janky project reporting - some code
We scripted GitHub for lightweight project reporting
My first DNF (Ft Bragg 600k)
It's been six years since my first ride with The San Francisco Randonneurs and four years since my first 200k. I've ridden 18 rides that are at least that distance since then (3x 300k, 2x 400k, 1x 600x), completing my first Super Randonneur Series (2-, 3-, 4-, and 600k in one year) last year after not riding much the year before that. And this weekend I had my first DNF result on the Fort Bragg 600k. I Did Not Finish.
The best response to my choice of abandoning the ride to enjoy the campground came from Peter Curley, who said "That was a very mature decision." A clear departure from typical randonneuring stubbornness and refusal to give up, I celebrated my decision to quit as a victory when I arrived at the campground and made my announcement to the volunteers. I think I was so energetic about it that they did not believe me. I was being kind to myself, to my body, and at peace with the decision by
(continued...)Should I Resign from My Full Professor Job to Work Fulltime on Cocalc?
Updated Statement About Our Relationship with DataCamp
We apologize for our poor communications about our response to the DataCamp sexual misconduct incident. We support the victims and we understand this has been a painful and ongoing struggle for them. We also recognize…
The post Updated Statement About Our Relationship with DataCamp appeared first on Anaconda.
PyCon2019 poem
I'm back in Cleveland for another Pycon. Yesterday was my first full day here. Along with Matt Seale, I was a helper at Matthias Bussonnier tutorial ("IPython and Jupyter in Depth: High productivity, interactive Python). The sticky system is efficient at signaling when someone in a classroom needs help, and a lot of folks don't know that this practice was popularized by Software Carpentry workshops and continues to be used at The Carpentries.
I stepped out for a coffee refill and bumped into a large contingent of Bloomberg folks I'd never met (Princeton office). I guess we have something like 90 people at the conference this year, and I made the usual and true remark about how I go to conferences to meet the other people who work at our company. Then after his tutorial concluded, Matthias and I bumped into Tracy Teal, exchanged some stickers, and chatted about The Carpentries, Jupyter, organizing conferences, governance and sponsorship models, and a bunch of other stuff.
Matthias was a
(continued...)Labs update and April highlights
It has been an exciting first month for me at Quansight Labs. It's a good time for a summary of what we worked on in April and what is coming next.
Progress on array computing librariesOur first bucket of activities I'd call "innovation". The most prominent
projects in this bucket are XND,
uarray,
metadsl,
python-moa,
Remote Backend Compiler and
arrayviews.
XND is an umbrella name for a set of related array
computing libraries: xnd, ndtypes, gumath, and xndtools.
Hameer Abbasi made some major steps forward with uarray: the backend and
coercion semantics are now largely worked out, there is
good documentation, and the
unumpy package (which currently has numpy, XND and PyTorch backends)
is progressing well. This blog post
gives a good overview of the motivation for uarray and its main concepts.
Saul Shanabrook and Chris Ostrouchov worked out how best to put metadsl
and python-moa together: metadsl can be used to create the API for
python-moa to simplify the code base of the latter a lot. Chris
also wrote an
Anaconda’s Response to DataCamp’s CEO and Board of Directors
DataCamp has been a business partner of our company for almost two years. So we were shocked and saddened by the recent allegations of inappropriate sexual behavior and retaliatory firings made against DataCamp’s CEO and…
The post Anaconda’s Response to DataCamp’s CEO and Board of Directors appeared first on Anaconda.
What's New in SymPy 1.4
As of November, 2018, I have been working at Quansight, under the heading of Quansight Labs. Quansight Labs is a public-benefit division of Quansight. It provides a home for a "PyData Core Team" which consists of developers, community managers, designers, and documentation writers who build open-source technology and grow open-source communities around all aspects of the AI and Data Science workflow. As a part of this, I am able to spend a fraction of my time working on SymPy. SymPy, for those who do not know, is a symbolic mathematics library written in pure Python. I am the lead maintainer of SymPy.
SymPy 1.4 was released on April 9, 2019. In this post, I'd like to go over some of the highlights for this release. The full release notes for the release can be found on the SymPy wiki.
To update to SymPy 1.4, use
conda install sympy
or if you prefer to use pip
pip install -U sympy
The SymPy 1.4 release contains over 500 changes from 38 different submodules, so I will not be
IBM Renews Corporate Sponsorship of NumFOCUS
The post IBM Renews Corporate Sponsorship of NumFOCUS appeared first on NumFOCUS.
DISC Committee Nominations are open for 2019-2021 positions
The post DISC Committee Nominations are open for 2019-2021 positions appeared first on NumFOCUS.
Reflections on AnacondaCON 2019 with NVIDIA’s Josh Patterson
I love this month. April 19’ brings back Game of Thrones, Avengers: Endgame (that Thanos snap though), and of course AnacondaCON. I’ve been to every AnacondaCON, which makes this my third show. Of all data…
The post Reflections on AnacondaCON 2019 with NVIDIA’s Josh Patterson appeared first on Anaconda.
uarray: A Generic Override Framework for Methods
uarray: A Generic Override Framework for Methods¶uarray is an override framework for methods in Python. In the scientific Python ecosystem, and in other similar places, there has been one recurring problem: That similar tools to do a job have existed, but don't conform to a single, well-defined API. uarray tries to solve this problem in general, but also for the scientific Python ecosystem in particular, by defining APIs independent of their implementations.
When SciPy was created, and Numeric and Numarray unified into NumPy, it jump-started Python's data science community. The ecosystem grew quickly: Academics started moving to SciPy, and the Scikits that popped up made the transition all the more smooth.
However, the scientific Python community also shifted during that time: GPUs and distributed computing emerged. Also, there were old ideas that couldn't really be used with NumPy's API, such as sparse arrays. To solve these problems, various libraries emerged:
- Dask, for distributed NumPy
- CuPy, for
Python Lambda Function with Examples
anonymous function as the function can be defined without a name. 
lambda arguments: expressionLambda function can have more than one argument but expression cannot be more than 1. The expression is evaluated and returned. Example
addition = lambda x,y: x + yIn the above python code,
addition(2,3) returns 5
x,y are the arguments and x + y is the expression that gets evaluated and returned. Difference between Lambda and Def FunctionBy using both lambda and def, you can create your own user-defined functionTensorFlow CPU optimizations in Anaconda
By Stan Seibert, Anaconda, Inc. & Nathan Greeneltch, Intel Corporation TensorFlow is one of the most commonly used frameworks for large-scale machine learning, especially deep learning (we’ll call it “DL” for short). This popular framework has…
The post TensorFlow CPU optimizations in Anaconda appeared first on Anaconda.
NumFOCUS Projects Accepted to Inaugural Google Season of Docs
The post NumFOCUS Projects Accepted to Inaugural Google Season of Docs appeared first on NumFOCUS.
Loops in Python explained with examples
Real World Examples of Loop
- Software of the ATM machine is in a loop to process transaction after transaction until you acknowledge that you have no more to do.
- Software program in a mobile device allows user to unlock the mobile with 5 password attempts. After that it resets mobile device.
- You put your favorite song on a repeat mode. It is also a loop.
- You want to run a particular analysis on each column of your data
MOA: a theory for composable and verifiable tensor computations
Python-moa (mathematics of arrays) is an approach to a high level tensor
compiler that is based on the work of Lenore
Mullin and her
dissertation.
A high level compiler is necessary because there are many optimizations
that a low level compiler such as gcc will miss. It is trying to solve
many of the same problems as other technologies such as the taco
compiler and the xla
compiler. However, it takes a much
different approach than others guided by the following principles.
- What is the shape? Everything has a shape. scalars, vectors, arrays, operations, and functions.
- What are the given indicies and operations required to produce a given index in the result?
Having a compiler that is guided upon these principles allows for high level reductions that other compilers will miss and allows for optimization of algorithms as a whole. Keep in mind that MOA is NOT a compiler. It is a theory that guides compiler development. Since python-moa is based on theory we get unique properties that other compilers cannot guarantee:
NumFOCUS Hires Executive Operations Administrator
The post NumFOCUS Hires Executive Operations Administrator appeared first on NumFOCUS.
Some questions and thoughts on journal peer review.
What's up with current peer review practice?
Create Dummy Data in Python
DataFrame functionimport pandas as pd
data = pd.DataFrame({"A" : ["John","Deep","Julia","Kate","Sandy"],
"MonthSales" : [25,30,35,40,45]})
A MonthSales
0 John 25
1 Deep 30
2
The Human Element in AI
The over 45 speakers at AnacondaCON 2019 delved into how machine learning, artificial intelligence, enterprise, and open source communities are accomplishing great things with data — from optimizing urban farming to identifying the elements in…
The post The Human Element in AI appeared first on Anaconda.
Things to think about when developing shotgun metagenome classifiers
Thoughts on goals and tradeoffs in classifying shotgun metagenome data.
Our submission for the NHGRI Human Genome Reference Center call
We wrote a big grant proposal!
PIP connection Error : SSL CERTIFICATE VERIFY FAILED
Error looks like this :
Could not fetch URL https://pypi.python.org/: connection error: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:598)
 |
| PIP SSL Certification Issue |
pip install --trusted-host pypi.org --trusted-host files.pythonhosted.org <package_name> -vvv
pip install --trusted-host pypi.org --trusted-host files.pythonhosted.org pandas -vvvDeepanshu founded ListenData with a simple objective - Make analytics easy to understand and follow. He has over 8 years of experience in
AnacondaCON 2019 Day 3 Recap: The Need for Speed, “Delightful UX” in Dev Tools, LOTR Jokes and More.
Everyone at Anaconda is still feeling the love AnacondaCON 2019. Day 3 wrapped up last Friday with one more day of talks and sessions, highlighted by some powerhouse keynotes. Let’s get right to the good…
The post AnacondaCON 2019 Day 3 Recap: The Need for Speed, “Delightful UX” in Dev Tools, LOTR Jokes and More. appeared first on Anaconda.
Install Python Package
Ways to Install Python Package
If Anaconda is installed on your system (laptop), click on Anaconda Prompt as shown in the image below. |
| Anaconda Prompt |
To install a python package or module, enter the code below in Anaconda Prompt -
pip install package-name
 |
| Install |
News from the NIH Data Commons Pilot Phase Consortium
The NIH Data Commons Pilot Phase Consortium is dead! (Long live the NIH Data Commons!)
NumFOCUS Releases 2018 Annual Report
The post NumFOCUS Releases 2018 Annual Report appeared first on NumFOCUS.
NumFOCUS Announces Cambridge Spark will Host a PyData Conference in 2019
The post NumFOCUS Announces Cambridge Spark will Host a PyData Conference in 2019 appeared first on NumFOCUS.
Critically assessing open science - the CAOS meeting.
A summary of the CAOS open science meeting
Anaconda 2019.03 Release
Windows is the most popular operating system in the world and consistently has 75% or more of the worldwide desktop market. According to the JetBrains Python Developers Survey, 49% of Python developers use Windows as…
The post Anaconda 2019.03 Release appeared first on Anaconda.
AnacondaCON 2019 Day 2 Recap: AI in Medicine, Cataloging the Contents of Stars, and More!
What You Missed at AnacondaCON Day 2 We’re back with a recap of Day 2 of our annual AnacondaCON. (In case you missed it, you can read our Day 1 recap here). Things started off…
The post AnacondaCON 2019 Day 2 Recap: AI in Medicine, Cataloging the Contents of Stars, and More! appeared first on Anaconda.
AnacondaCON 2019 Day 1 Recap: Big-Time Learning
AnacondaCON 2019 is off to a great start. As in past years, we programmed Day 1 with product- and package-specific tutorials for those looking to get hands-on learning with Anaconda Enterprise tools. Spots in these…
The post AnacondaCON 2019 Day 1 Recap: Big-Time Learning appeared first on Anaconda.
3 Ways to Upskill in Python with DataCamp and Anaconda
DataCamp is proud to partner with Anaconda to offer eight courses on Conda and Python—in addition to the more than 70 total Python courses in DataCamp’s ever-expanding data science and analytics curriculum. Not sure where…
The post 3 Ways to Upskill in Python with DataCamp and Anaconda appeared first on Anaconda.
Facebook joins NumFOCUS Corporate Sponsors
The post Facebook joins NumFOCUS Corporate Sponsors appeared first on NumFOCUS.
NumFOCUS Welcomes Sam Brice as inaugural Visiting Fellow
The post NumFOCUS Welcomes Sam Brice as inaugural Visiting Fellow appeared first on NumFOCUS.
Computational Psychiatry: Combining multiple levels of analysis to understand brain disorders - PhD thesis
I noticed that as my personal website at my former university went down that my PhD thesis could not be found anywhere, so I'm posting it here.
During my PhD I explored how machine learning and computational modeling of the brain can be used to improve our understanding, and diagnostics …
NIXML: nix + YAML for easy reproducible environments
The rise and fall of bioconda A year ago, I remember a conversation which went basically like this: Them: So, to distribute my package, what do you think I should use? Me: You should use bioconda. Them: OK, that’s interesting, but what about …? Me: No, you should use bioconda. Them: I will definitely look … Continue reading NIXML: nix + YAML for easy reproducible environments →NumFOCUS Hires New Staff to Support Events and Financial Services
The post NumFOCUS Hires New Staff to Support Events and Financial Services appeared first on NumFOCUS.
Sustaining open source: thinking about communities of effort
Thinking about how to sustain open source.
My recent reading re sustaining open communities
What has Titus been reading lately?
Blosc joins NumFOCUS Sponsored Projects
The post Blosc joins NumFOCUS Sponsored Projects appeared first on NumFOCUS.
Gina Helfrich on the Changelog Podcast
The post Gina Helfrich on the Changelog Podcast appeared first on NumFOCUS.
Reduzindo a pilha
Sou fã de quadrinhos desde criança. As primeiras revistas que ganhei foram na primeira metade dos anos 90, alguns Mickeys, Mônicas, Trapalhões e X-Men. Em 98 comecei a comprar X-Men, Fabulosos X-Men e Wolverine, até os primeiros números da famigerada X-Men Premium. Sem dinheiro, enveredei pelos mangás e histórias fechadas. Quando a Panini começa a... [Read More]Threat models for open online scientific engagement?
What threats are there for scientists in engaging in open online discussions?
Packaging PyQt5 apps with fbs — Distribute cross-platform GUI applications with the fman Build System
fbs is a cross-platform PyQt5 packaging system which supports building desktop applications for Windows, Mac and Linux (Ubuntu, Fedora and Arch). Built on top of PyInstaller it wraps some of the rough edges and defines a standard project structure which allows the build process to be entirely automated. The included …
Announcement: Audio TK 3.1.0
ATK is updated to 3.1.0 with heavy code refactoring. Old C++ standards are now dropped and it requires now a full C++17 compliant compiler. The main difference for filter support is that explicit SIMD filters using libsimdpp have been dropped while tr2::simd becomes standard and supported by gcc, clang and Visual Studio. Download link: ATK […]Telemulta
A voz feminina robótica (chegamos no tempo onde questão de gênero e robôs podem se confundir) soou, estranha e familiar como sempre, assim que o carro finalizou a curva para a direita: “Você entrou na Avenida Universitária; o limite de velocidade é 60 quilômetros por hora”. Meu pai sorriu e começou a falar: – Desde... [Read More]My foreword to "Bayesian Analysis with Python, 2nd Edition" by Osvaldo Martin
When Osvaldo asked me to write the foreword to his new book I felt honored, excited, and a bit scared, so naturally I accepted. What follows is my best attempt to convey what makes probabilistic programming so exciting to me. Osvaldo did a great job with the book, it is …
Call for Answers: Survey About Task Assignment
Professor Igor Steinmacher, from Northern Arizona University, is a proeminent researcher on several social dynamics in open source communities, like support of newcomers, gender bias, open sourcing proprietary software, and more. Some of his papers can de found in his website. Currently, Prof. Igor is inviting mentors from open source communities to answer a survey... [Read More]Revisiting authorship, and JOSS software publications
The question du jour: how should authorship on software papers be decided?
Using Bayesian Decision Making to Optimize Supply Chains
(c) 2019 Thomas Wiecki & Ravin Kumar
As advocates of Bayesian statistics in data science we often have to convince business-minded colleagues or customers of the added value of such an approach. While there are many good reasons for applying Bayesian modeling to solve business problems (Sean J Taylor recently had …
Mestrado em Ciência da Computação na UFPA 2019: Inteligência Computacional para Smart Grids; Metaheurísticas
Está aberto o processo seletivo para o mestrado em ciência da computação do PPGCC-UFPA. Nesse certame, estou disponibilizando 2 vagas para alunos que desenvolverão seus trabalhos junto aos demais pesquisadores no LAAI. As vagas são voltadas para os temas de inteligência computacional aplicada a Smart Grids e estudos sobre métodos metaheurísticos de otimização. Gostaria de... [Read More]LaKademy 2018
Em outubro de 2018, Florianópolis foi sede da sexta edição do LaKademy, o sprint latinoamericano do KDE. Esse momento é uma oportunidade para termos em um mesmo lugar vários desenvolvedores do KDE – tanto veteranos quanto novatos – de diferentes projetos para melhorarem os respectivos softwares em que trabalham e também planejar as ações de... [Read More]LaKademy 2018
Past October 2018, Florianópolis hosted the 6th edition of LaKademy, the Latin-American KDE sprint. That moment is an opportunity to put together several KDE developers – both veterans and newcomers – from different projects in order to work for improve their respective software and plan the promotional actions of the community in the subcontinent. In... [Read More]GPU Dask Arrays, first steps
The following code creates and manipulates 2 TB of randomly generated data.
import dask.array as da
rs = da.random.RandomState()
x = rs.normal(10, 1, size=(500000, 500000), chunks=(10000, 10000))
(x + 1)[::2, ::2].sum().compute(scheduler='threads')
On a single CPU, this computation takes two hours.
On an eight-GPU single-node system this computation takes nineteen seconds.
Combine Dask Array with CuPyActually this computation isn’t that impressive. It’s a simple workload, for which most of the time is spent creating and destroying random data. The computation and communication patterns are simple, reflecting the simplicity commonly found in data processing workloads.
What is impressive is that we were able to create a distributed parallel GPU array quickly by composing these three existing libraries:
-
CuPy provides a partial implementation of Numpy on the GPU.
-
Dask Array provides chunked algorithms on top of Numpy-like libraries like Numpy and CuPy.
This enables us to operate on more data than we could fit in memory by operating on that data in
Manage project dependencies with conda environments
TL;DR: Create a conda environment for each project, capture exact versions when possible, automate activation and updating with a bash function.
I often work on several different projects involving software: Python libraries, papers, presentations, posters, this website, etc. Each project has different dependencies and there is a non-zero chance that these dependencies might be in conflict with each other. For example, I need Python 2.7 to work on a tesseroid modeling paper with a student, while my current work on
(continued...)First Impressions of GPUs and PyData
I recently moved from Anaconda to NVIDIA within the RAPIDS team, which is building a PyData-friendly GPU-enabled data science stack. For my first week I explored some of the current challenges of working with GPUs in the PyData ecosystem. This post shares my first impressions and also outlines plans for near-term work.
First, lets start with the value proposition of GPUs, significant speed increases over traditional CPUs.
GPU PerformanceLike many PyData developers, I’m loosely aware that GPUs are sometimes fast, but don’t deal with them often enough to have strong feeling about them.
To get a more visceral feel for the performance differences, I logged into a GPU machine, opened up CuPy (a Numpy-like GPU library developed mostly by Chainer in Japan) and cuDF (a Pandas-like library in development at NVIDIA) and did a couple of small speed comparisons:
Compare Numpy and Cupy>>> import numpy, cupy
>>> x = numpy.random.random((10000, 10000))
>>> y = cupy.random.random((10000, 10000))
>>> %timeit bool((numpy.sin(x) ** 2 + numpy.cos(x) ** 2 == 1).all())
446 ms ± 53.1 ms perA quick read of _The genomic and proteomic landscape of the rumen microbiome_
Using short and long reads to assemble genomes from metagenomes!
Support Python 2 with Cython
SummaryMany popular Python packages are dropping support for Python 2 next month. This will be painful for several large institutions. Cython can provide a temporary fix by letting us compile a Python 3 codebase into something usable by Python 2 in many cases.
It’s not clear if we should do this, but it’s an interesting and little known feature of Cython.
Background: Dropping Python 2 Might be Harder than we ExpectMany major numeric Python packages are dropping support for Python 2 at the end of this year. This includes packages like Numpy, Pandas, and Scikit-Learn. Jupyter already dropped Python 2 earlier this year.
For most developers in the ecosystem this isn’t a problem. Most of our packages are Python-3 compatible and we’ve learned how to switch libraries. However, for larger companies or government organizations it’s often far harder to switch. The PyCon 2017 keynote by Lisa Guo and Hui Ding from Instagram gives a good look into why this can be challenging for large production codebases and also gives a good
(continued...)Anatomy of an OSS Institutional Visit
I recently visited the UK Meteorology Office, a moderately large organization that serves the weather and climate forecasting needs of the UK (and several other nations). I was there with other open source colleagues including Joe Hamman and Ryan May from open source projects like Dask, Xarray, JupyterHub, MetPy, Cartopy, and the broader Pangeo community.
This visit was like many other visits I’ve had over the years that are centered around showing open source tooling to large institutions, so I thought I’d write about it in hopes that it helps other people in this situation in the future.
My goals for these visits are the following:
- Teach the institution about software projects and approaches that may help them to have a more positive impact on the world
- Engage them in those software projects and hopefully spread around the maintenance and feature development burden a bit
We were invited by early adopters within the institution, both within the UK Met Office’s Informatics Lab
(continued...)The story behind the spacegraphcats project and paper
The TRUE story!
Notes on the Frank-Wolfe Algorithm, Part II: A Primal-dual Analysis
This blog post extends the convergence theory from the first part of my notes on the Frank-Wolfe (FW) algorithm with convergence guarantees on the primal-dual gap which generalize and strengthen the convergence guarantees obtained in the first part.
MathJax.Hub.Config({ extensions: ["tex2jax.js"], jax: ["input/TeX", "output/HTML-CSS"], tex2jax …Creating a welcoming teaching/learning environment in workshops
It takes constant work to make a welcoming teaching/learning environment!
Repeatability in Practice (2018 version)
How we do repeatability in the DIB Lab
Linking to emails in org-mode (using neomutt)
Where we store links to emails in org-mode, and open them using neomutt.
Ode ao ódio
Ontem, acompanhando a apuração para presidente no 2º turno, chorei. Chorei de raiva. Chorei de ódio. Ódio porque aquele que levou o pleito representa uma total afronta ao mínimo do que chamamos civilidade. Ele defendeu a ditadura e a tortura, reiteradamente. Prometeu prender ou exilar opositores. Prometeu perseguir professores, artistas, a intelectualidade. Disse que irá... [Read More]Eleições 2018: Minha carta para a família
Família, essa é minha última manifestação política aqui no grupo antes do resultado. Vocês me conhecem, sou professor de ciência da computação na UFPA, sou um dos responsáveis pela formação dos próximos engenheiros de software e matemáticos computacionais da nossa região. Oriento alunos na graduação, no mestrado e também no doutorado, mesmo com todas as... [Read More]A arquitetura de compartilhamentos do Telegram para mitigar as fake news no WhatsApp
Fake News já se tornaram o tipo de problema que teremos que enfrentar de alguma maneira o quanto antes, ou veremos democracias sendo destruídas uma a uma. Se o caso Trump nos chamava atenção mas ainda parecia distante, as eleições brasileiras de 2018 vieram pra mostrar que o tiozão gente boa pode se converter no... [Read More]So you want to contribute to open source
Welcome new open source contributor!
I appreciated receiving the e-mail where you said you were excited about getting into open source and were particularly interested in working on a project that I maintain. This post has a few thoughts on the topic.
First, please forgive me for sending you to this post rather than responding with a personal e-mail. Your situation is common today, so I thought I’d write up thoughts in a public place, rather than respond personally.
This post has two parts:
- Some pragmatic steps on how to get started
- A personal recommendation to think twice about where you focus your time
Most open source software (OSS) projects have a “Good first issue” label on their Github issue tracker. Here is a screenshot of how to find the “good first issue” label on the Pandas project:
(note that this may be named something else like “Easy to fix”)
This contains a list of issues that are important, but also
(continued...)Dictionary Views & Set Operations — Working with dictionary view objects
The keys, values and items from a dictionary can be accessed using the .keys(), .values() and .items() methods. These methods return view objects which provide a view on the source dictionary.
The view objects dict_keys and dict_items support set-like operations (the latter only when all values are hashable) which …
Ciro em frente!
Faltando poucos dias para o 1º turno das eleições, aproveito o momento para declarar meu voto em Ciro Gomes e convido amigos e amigas a ponderarem e também votarem no candidato. Em um conceito bastante generoso de partidos políticos, tratam-se de organizações estruturadas em torno de uma ideia de ordenamento social e que tentam, através... [Read More]Announcement: Audio TK 3.0.0
ATK is updated to 3.0.0 with a major ABI break and code quality improvement (see here). Bugs in different areas were fixed. Development for additional modules was also simplified (the modelling lite is such a project based on Audio Toolkit). Download link: ATK 3.0.0 Changelog: 3.0.0 * Change size for gsl::index everywhere (change of ABI) […]QtConsole 4.4 Released!
We're excited to announce a significant update to QtConsole—the package that powers Spyder's IPython Console interface—which the Spyder team maintains in collaboration with Project Jupyter.
Two of the biggest changes—user-selectable syntax highlighting themes, and enhanced external editor/IDE integration—are already built right into Spyder, so they'll likely be of more interest if you use QtConsole standalone or with another editor/IDE.
However, most of the other changes should prove quite useful within Spyder as well, and many were in fact suggested and even implemented by users of our IDE.
Particular highlights include a block indent/unindent feature, Select-All (Ctrl-Shift-A) being made cell-specific, Ctrl-Backspace and Ctrl-Delete behaving more intelligently across whitespace and line boundaries, Ctrl-D allowing you to easily exit ipdb, input() and the like, and numerous smaller enhancements and bug fixes.
If you'd like to learn more about what's new, please check out our article over on the Jupyter blog, where we go over the major changes in more detail, with plenty
Dask Development Log
This work is supported by Anaconda Inc
To increase transparency I’m trying to blog more often about the current work going on around Dask and related projects. Nothing here is ready for production. This blogpost is written in haste, so refined polish should not be expected.
Since the last update in the 0.19.0 release blogpost two weeks ago we’ve seen activity in the following areas:
- Update Dask examples to use JupyterLab on Binder
- Render Dask examples into static HTML pages for easier viewing
- Consolidate and unify disparate documentation
- Retire the hdfs3 library in favor of the solution in Apache Arrow.
- Continue work on hyper-parameter selection for incrementally trained models
- Publish two small bugfix releases
- Blogpost from the Pangeo community about combining Binder with Dask
- Skein/Yarn Update
The new dask-labextension embeds Dask’s dashboard plots into a JupyterLab session so that you can get easy access to information
(continued...)A foundation for scikit-learn at Inria
We have just announced that a foundation will be supporting scikit-learn at Inria [1]: scikit-learn.fondation-inria.fr
This is an exciting turn for us, because it enables us to receive private funding. As a result, we will be able to have secure employment for some existing core …
Introducing Verde
Verde is a Python library for processing spatial data (bathymetry, geophysics surveys, etc) and interpolating it on regular grids (i.e., gridding).
It implements Green's functions
based interpolation methods and other data processing routines.
The type of gridding implemented in Verde is essentially fitting various linear models
to spatial data and using them to predict new data on regular grids, which is what a lot
of machine learning is all about.
So Verde's gridder API
is inspired on scikit-learn, the state-of-the-art for
machine learning in Python.
The Green's functions that make up the Jacobian matrix (aka sensitivity or feature
matrix) of the linear models generally come from elastic deformation theory.
For example, the bi-harmonic spline (Sandwell, 1987)
implemented in verde.Spline comes from the deformation of a thin elastic plate.
I submitted a
(continued...)The Waiting Time Paradox, or, Why Is My Bus Always Late?
Image Source: Wikipedia License CC-BY-SA 3.0
If you, like me, frequently commute via public transit, you may be familiar with the following situation:
You arrive at the bus stop, ready to catch your bus: a line that advertises arrivals every 10 minutes. You glance at your watch and note the time... and when the bus finally comes 11 minutes later, you wonder why you always seem to be so unlucky.
Naïvely, you might expect that if buses are coming every 10 minutes and you arrive at a random time, your average wait would be something like 5 minutes. In reality, though, buses do not arrive exactly on schedule, and so you might wait longer. It turns out that under some reasonable assumptions, you can reach a startling conclusion:
When waiting for a bus that comes on average every 10 minutes, your average waiting time will be 10 minutes.
This is what is sometimes known as the waiting time paradox.
I've encountered this idea before, and always wondered
(continued...)Three Operator Splitting
I discuss a recently proposed optimization algorithm: the Davis-Yin three operator splitting.Dask Release 0.19.0
This work is supported by Anaconda Inc.
I’m pleased to announce the release of Dask version 0.19.0. This is a major release with bug fixes and new features. The last release was 0.18.2 on July 23rd. This blogpost outlines notable changes since the last release blogpost for 0.18.0 on June 14th.
You can conda install Dask:
conda install dask
or pip install from PyPI:
pip install dask[complete] --upgrade
Full changelogs are available here:
Notable ChangesA ton of work has happened over the past two months, but most of the changes are small and diffuse. Stability, feature parity with upstream libraries (like Numpy and Pandas), and performance have all significantly improved, but in ways that are difficult to condense into blogpost form.
That being said, here are a few of the more exciting changes in the new release.
Python VersionsWe’ve dropped official support for Python 3.4 and added official support for Python 3.7.
Deploy on Hadoop ClustersOver the past few months Jim Crist has bulit a suite of
(continued...)Book: Building Machine Learning Systems with Python – third edition
A few year ago, Packt Publishing contacted to be a technical reviewer for the first edition of Building Machine Learning Systems with Python, and I was impressed by the writing of Luis Pedro Coelho and Willi Richert. For the second edition, I was again a technical reviewer. Writing is not easy, especially when it’s not […]Abstract for SIAM: Supporting and Sustaining Open Source Software Development: the Commons Perspective
How do we support and sustain open source software development?
Analog modelling: The Moog ladder filter emulation in Python
After my previous post on SPICE modelling in Python, I need to use a good support example to go up to on the fly compilation in C++. This schema will also require some changes to support more than simple nodal analysis, so this now becomes Modified Nodal Analysis with state equations. The simple model I […]Dictionaries — An rather long guide to Python's key:value store
Dictionaries are key-value stores, meaning they store, and allow retrieval of data (or values) through a unique key. This is analogous with a real dictionary where you look up definitions (data) using a given key — the word. Unlike a language dictionary however, keys in Python dictionaries are not alphabetically sorted …
Can bits be the basis for a digital commons? (No.)
Bits cannot be the basis for a digital commons, because they are not rivalrous.
Spyder 3.3.0 and 3.3.1 released!
We're pleased to release the next significant update in the stable Spyder 3 line, 3.3.0, along with its follow-on bugfix point release, 3.3.1, which is now live on PyPI and conda.
As always, you can update with conda update spyder in the Anaconda Prompt/Terminal/command line (on Windows/macOS/Linux, respectively) if on Anaconda (recommended), or pip update spyder otherwise.
If you run into any trouble, please carefully read our new installation documentation and consult our Troubleshooting Guide, which contains straightforward solutions to the vast majority of install-related issues users have reported.
As a new minor version (3.3), it makes several substantial changes to Spyder's underpinnings that deserve some explanation, particularly the newly modular and portable console system that's now separated into its own spyder-kernels package, opening up several new options for users running Spyder in different environments.
There's also a brand-new error reporting process, new options in the IPython console, usability and performance improvements for the Variable Explorer, multiple new and changed dependency requirements
Hierarchical Bayesian Neural Networks with Informative Priors
(c) 2018 by Thomas Wiecki
Imagine you have a machine learning (ML) problem but only small data (gasp, yes, this does exist). This often happens when your data set is nested -- you might have many data points, but only few per category. For example, in ad-tech you may want predict …
Spyder featured on Episode 1 of Open Source Directions web show
Quansight, the company recently founded by NumPy, SciPy and Anaconda creator Travis Oliphant to help connect companies with open source communities built around data science and machine learning, just released Episode 1 of its live webcast series, and it was all about Spyder! Spyder maintainer Carlos Córdoba, recently hired by Quansight and funded part-time to work on Spyder development as we announced a few weeks ago, was the featured guest on the show.
Carlos first shared his perspective on some of the key moments in Spyder's nearly 10-year development history, from its original creation by Pierre Raybaut and Carlos' initial involvement in the project to its more recent challenges and successes. He also demonstrated basic usage of Spyder, as well as some of its standout features, in a live on-screen demo. Carlos then went on to outline the current roadmap for Spyder 4 in the near future, and explained some of the key new features planned for it. Finally, he took
(continued...)