Planet SciPy
What Can AI Teach Us about Bias and Fairness?
By: Peter Wang & Natalie Parra-Novosad As researchers, journalists, and many others have discovered, machine learning algorithms can deliver biased results. One notorious example is ProPublica’s discovery of bias in a software called COMPAS used…
The post What Can AI Teach Us about Bias and Fairness? appeared first on Anaconda.
Announcing Anaconda Distribution 2019.10
We are pleased to announce the release of Anaconda Distribution 2019.10! As there were some significant changes in the previous Anaconda Distribution 2019.07 installers, this release focuses on polishing up rough edges in that release…
The post Announcing Anaconda Distribution 2019.10 appeared first on Anaconda.
O SERPRO e a validação de documentos digitais
No rascunho do post anterior sobre os documentos digitais no Brasil acabei escrevendo bastante sobre o papel do SERPRO nesse processo – tanto que decidi separá-lo em um post próprio. Com o lançamento do e-Título foi necessário para o TSE criar uma maneira de validar o documento digital para evitar fraudes. A tecnologia adotada foi… Continue a ler »O SERPRO e a validação de documentos digitaisHow to Restore Anaconda after Update to MacOS Catalina
MacOS Catalina was released on October 7, 2019, and has been causing quite a stir for Anaconda users. Apple has decided that Anaconda’s default install location in the root folder is not allowed. It moves…
The post How to Restore Anaconda after Update to MacOS Catalina appeared first on Anaconda.
Anaconda Enters a New Chapter
Today I am excited to announce that I am stepping into the role of CEO at Anaconda. Although I am a founder of the company and have previously served as president, this marks the first…
The post Anaconda Enters a New Chapter appeared first on Anaconda.
Quansight Labs Work Update for September, 2019
As of November, 2018, I have been working at Quansight. Quansight is a new startup founded by the same people who started Anaconda, which aims to connect companies and open source communities, and offers consulting, training, support and mentoring services. I work under the heading of Quansight Labs. Quansight Labs is a public-benefit division of Quansight. It provides a home for a "PyData Core Team" which consists of developers, community managers, designers, and documentation writers who build open-source technology and grow open-source communities around all aspects of the AI and Data Science workflow.
My work at Quansight is split between doing open source consulting for various companies, and working on SymPy. SymPy, for those who do not know, is a symbolic mathematics library written in pure Python. I am the lead maintainer of SymPy.
In this post, I will detail some of the open source work that I have done recently, both as part of my open source consulting, and as part of my work on SymPy for Quansight Labs.
Bounds Checking in NumbaAs part
Os documentos digitais (no plural) do Brasil
Já faz algum tempo o Brasil está passando por um processo de digitalização dos documentos oficiais utilizados por pessoas físicas. Entretanto, o que antes se anunciava como uma possível convergência dos mais diferentes documentos para um documento único, que serviria para tudo, passou a acontecer a digitalização de cada documento específico através do desenvolvimento de… Continue a ler »Os documentos digitais (no plural) do BrasilHow to Evaluate the Logistic Loss and not NaN trying
A naive implementation of the logistic regression loss can results in numerical indeterminacy even for moderate values. This post takes a closer look into the source of these instabilities and discusses more robust Python implementations.
hljs.initHighlightingOnLoad(); MathJax.Hub.Config({ extensions: ["tex2jax.js"], jax: ["input/TeX", "output/HTML-CSS"], tex2jax: { inlineMath …Anaconda Enterprise Receives Honors in Fourth Annual Datanami Readers’ and Editors’ Choice Awards
SAN DIEGO, Sept. 23, 2019 — Anaconda’s enterprise data science platform has been recognized in the fourth annual Datanami Readers’ and Editors’ Choice Awards, presented during the Strata Data Conference. The list of winners was…
The post Anaconda Enterprise Receives Honors in Fourth Annual Datanami Readers’ and Editors’ Choice Awards appeared first on Anaconda.
Uvas Gold 200
My poem about a rainy 200k was published in the Fall 2019 issue of American Randonneur (a quarterly magazine published by Randonneurs USA)
I've been doing samizdat poetry for as long as I've had a web presence (since 1999), but I am now officially a published poet! (I am deliberately not counting the embarrasing hackjob that was published in a youth anthology when I was in 8th grade.)
You can find "Uvas Gold 200" on page 26 - either directly on this skeuomorphic leafing viewer or the PDF, but I'm republishing both the exposition blurb and the poem below. If you prefer to listen, I recorded a reading of it that you can download in different flavors: a local audio only, a local video, or the embeded video version below.
(continued...)Uvas Gold 200k starts and ends in Fremont, CA and was held on Saturday, December 1st, 2018. The ride frontloads the climbing by going nearly half-way up Mount
Introducing Our Newest Corporate Sponsorship Prospectus
The post Introducing Our Newest Corporate Sponsorship Prospectus appeared first on NumFOCUS.
Grupo de Estudos do Laboratório Amazônico de Estudos Sociotécnicos – UFPA
Eu e o prof. Leonardo Cruz da Faculdade de Ciências Sociais estamos juntos trabalhando no desenvolvimento do Laboratório Amazônico de Estudos Sociotécnicos da UFPA. Nossa proposta é realizar leituras e debates críticos sobre o tema da sociologia da tecnologia, produzir pesquisas teóricas e empíricas na região amazônica sobre as relações entre tecnologia e sociedade, e… Continue a ler »Grupo de Estudos do Laboratório Amazônico de Estudos Sociotécnicos – UFPARuby wrappers for the XND project
Table of Contents
IntroductionLack of stable and reliable scientific computing software has been a persistent problem for the Ruby community, making it hard for enthusiastic Ruby developers to use Ruby in everything from their web applications to their data analysis projects. One of the most important components of any successful scientific software stack is a well maintained and flexible array computation library that can act as a fast and simple way of storing in-memory data and interfacing it with various fast and battle-tested libraries like LAPACK and BLAS.
Various projects have attempted to make such libraries in the past (and some are still thriving and maintained). Some of the notable ones are numo, nmatrix, and more recently, numruby. These projects attempt
Highlights From The 2019 Pandas Hack
The post Highlights From The 2019 Pandas Hack appeared first on NumFOCUS.
Machine Learning in Healthcare: 5 Use Cases that Improve Patient Outcomes
Machine learning is accelerating the pace of scientific discovery across fields, and medicine is no exception. From language processing tools that accelerate research to predictive algorithms that alert medical staff of an impending heart attack,…
The post Machine Learning in Healthcare: 5 Use Cases that Improve Patient Outcomes appeared first on Anaconda.
QuantumBlack joins NumFOCUS as Corporate Sponsor
The post QuantumBlack joins NumFOCUS as Corporate Sponsor appeared first on NumFOCUS.
Canaries Can Tweet: Preview New Features with Conda Canary
Conda-canary is the pre-defaults-release channel for conda — it has the most recent version of conda. On occasion it will also have the latest pre-defaults-release of conda-build and other conda dependencies such as ruamel.yaml. Normally,…
The post Canaries Can Tweet: Preview New Features with Conda Canary appeared first on Anaconda.
Quansight Labs Dask Update
This post provides an update on some recent Dask-related activities the Quansight Labs team has been working on.
Dask community work orderThrough a community work order (CWO) with the D. E. Shaw group, the Quansight Labs team has been able to dedicate developer time towards bug fixes and feature requests for Dask. This work has touched on several portions of the Dask codebase, but generally have centered around using Dask Arrays with the distributed scheduler.
Read more… (2 min remaining to read)
How to Build a Custom Anaconda Installer for R
A frequent question on the Anaconda Community mailing list is how to package R with conda for distribution. Depending on the use case, one option may be to use conda to move environments. This requires…
The post How to Build a Custom Anaconda Installer for R appeared first on Anaconda.
pelas ruas de Belém
as vezes pelas ruas de Belém não sei se sou eu ou minha mãe quem está aliEnterprises Need to Think Differently about Data Science. Here’s How.
Companies that are data science literate make and communicate decisions on the basis of real data models, and not merely instinct or tradition. They welcome new data science technologies as opportunities for potential innovation, rather…
The post Enterprises Need to Think Differently about Data Science. Here’s How. appeared first on Anaconda.
Moving Conda Environments
Conda is known as a package manager for Python and R packages produced by Anaconda, Inc. and conda-forge, the open-source community for conda Python packages. In addition to managing packages, Conda is also an environment…
The post Moving Conda Environments appeared first on Anaconda.
Spyder 4.0 beta4: Kite integration is here
Kite is sponsoring the work discussed in this blog post, and in addition supports Spyder 4.0 development through a Quansight Labs Community Work Order.
As part of our next release, we are proud to announce an additional completion client for Spyder, Kite. Kite is a novel completion client that uses Machine Learning techniques to find and predict the best autocompletion for a given text. Additionally, it collects improved documentation for compiled packages, i.e., Matplotlib, NumPy, SciPy that cannot be obtained easily by using traditional code analysis packages such as Jedi.
Read more… (3 min remaining to read)
4 Machine Learning Use Cases in the Automotive Sector
From parts suppliers to vehicle manufacturers, service providers to rental car companies, the automotive and related mobility industries stand to gain significantly from implementing machine learning at scale. We see the big automakers investing in…
The post 4 Machine Learning Use Cases in the Automotive Sector appeared first on Anaconda.
An initial report on the Common Fund Data Ecosystem
Helping DCCs get FAIRer
Accessing Remote Data with a Generalized File System
Originally created for the needs of Dask, we have spun out a general file system implementation and specification, to provide all users with simple access to many local, cluster, and remote storage media. Dask and Intake…
The post Accessing Remote Data with a Generalized File System appeared first on Anaconda.
Object Oriented Programming in Python : Learn by Examples
In the above example, car is an object. Functions are called methods in OOP world. Characteristics are attributes (properties). Technically attributes are variables or values related to the state of the object whereas methods
Dask joins NumFOCUS Sponsored Projects
The post Dask joins NumFOCUS Sponsored Projects appeared first on NumFOCUS.
Precision Recall Curve Simplified
Meet our 2019 Google Summer of Code Students (Part 3)
The post Meet our 2019 Google Summer of Code Students (Part 3) appeared first on NumFOCUS.
Meet our 2019 Google Summer of Code Students (Part 2)
The post Meet our 2019 Google Summer of Code Students (Part 2) appeared first on NumFOCUS.
Comparing two genome binnings quickly with sourmash
Comparing two sets of MAGs, for fun and profit!
Calculate KS Statistic with Python
In data science, it compares the cumulative distribution of events and non-events and KS is where there is a maximum difference between the two distributions. In simple words, it helps us to understand how well our predictive model is able to discriminate between events and
A Complete Guide to Python DateTime Functions
- datetime
- date
- time
- timedelta
People who have no experience of working with real-world datasets might have not encountered date columns. They might be under impression that working with dates is rarely used and not so
Meet our 2019 Google Summer of Code Students (Part 1)
The post Meet our 2019 Google Summer of Code Students (Part 1) appeared first on NumFOCUS.
What are *args and **kwargs and How to use them
args is a short form of arguments. With the use of *args python takes any number of arguments in user-defined function and converts user inputs to a tuple named args. In other words, *args means zero or more arguments which are stored in a tuple named args. When you define function without *args, it has a fixed number of inputs which means it cannot accept more (or less) arguments than you defined in the function.
In the example code below, we are creating a very basic function which adds two numbers. At the same time, we created a
Quansight presence at SciPy'19
Yesterday the SciPy'19 conference ended. It was a lot of fun, and very productive. You can really feel that there's a lot of energy in the community, and that it's growing and maturing. This post is just a quick update to summarize Quansight's presence and contributions, as well as some of the more interesting things I noticed.
A few highlightsThe "Open Source Communities" track, which had a strong emphasis on topics like burnout, diversity and sustainability, as well as the keynotes by Stuart Geiger ("The Invisible Work of Maintaining and Sustaining Open-Source Software") and Carol Willing ("Jupyter: Always Open for Learning and Discovery") showed that many more people and projects are paying more attention to and evolving their thinking on the human and organizational aspects of open source.
I did not go to many technical talks, but did make sure to catch Matt Rocklin's talk "Refactoring the SciPy Ecosystem for Heterogeneous Computing". Matt clearly explained some key issues and opportunities around
Python : 10 Ways to Filter Pandas DataFrame
- Select all the active customers whose accounts were opened
Ibis: Python data analysis productivity framework
Ibis is a library pretty useful on data analysis tasks that provides a pandas-like API that allows operations like create filter, add columns, apply math operations etc in a lazy mode so all the operations are just registered in memory but not executed and when you want to get the result of the expression you created, Ibis compiles that and makes a request to the remote server (remote storage and execution systems like Hadoop components or SQL databases). Its goal is to simplify analytical workflows and make you more productive.
Ibis was created by Wes McKinney and is mainly maintained by Phillip Cloud and Krisztián Szűcs. Also, recently, I was invited to become a maintainer of the Ibis repository!
Maybe you are thinking: "why should I use Ibis?". Well, if you have any of the following issues, probably you should consider using Ibis in your analytical workflow!
- if you need to get data from a SQL database but you don't
Python Dictionary Comprehension with Examples
- Keys : Think about columns in tables. It must be unique (like column names cannot be duplicate)
- Values : It is similar to rows in tables. It can be duplicate.
To extract keys, values and structure of dictionary, you can submit the following commands.
d = {'a': [1,2], 'b': [3,4], 'c': [5,6]}
d.keys() # 'a', 'b', 'c'
d.values() # [1, 2], [3, 4], [5,
HTML outputs in Jupyter
SummaryUser interaction in data science projects can be improved by adding a small amount of visual deisgn.
To motivate effort around visual design we show several simple-yet-useful examples. The code behind these examples is small and accessible to most Python developers, even if they don’t have much HTML experience.
This post in particular focuses on Jupyter’s ability to add HTML output to any object. This can either be full-fledged interactive widgets, or just rich static outputs like tables or diagrams. We hope that by showing examples here we will inspire some throughts in other projects.
This post was supported by replies to this tweet. The rest of this post is just examples.
IrisI originally decided to write this post after reading another blogpost from the UK Met office, where they included the HTML output of their library Iris in a a blogpost
(work by Peter Killick, post by Theo McCaie)
The fact that the output provided by an interactive session is the same output that you would provide in a published result helps everyone. The interactive
(continued...)Python list comprehension with Examples
f(x) = x2, f(x) will always return the same result with the same x value. The function has no "side-effect" which means an operation has no effect on a variable/object that is outside the intended usage. "Side-effect" refers to leaks in your code which can modify a mutable data structure or variable. Functional programming is also good for parallel computing as there is no
uarray update: API changes, overhead and comparison to __array_function__
uarray is a generic override framework for objects and methods in Python. Since my last uarray blogpost, there have been plenty of developments, changes to the API and improvements to the overhead of the protocol. Let’s begin with a walk-through of the current feature set and API, and then move on to current developments and how it compares to __array_function__. For further details on the API and latest developments, please see the API page for uarray. The examples there are doctested, so they will always be current.
NumPy is a simple, rectangular, dense, and in-memory data store. This is great for some applications but isn't complete on its own. It doesn't encompass every single use-case. The following are examples of array objects available today that have different features and cater to a different kind of audience.
- Dask is one of the most popular ones. It allows distributed and chunked computation.
- CuPy is another popular one, and
Don't cite the No Free Lunch Theorem
Tldr; You probably shouldn’t be citing the "No Free Lunch" Theorem by Wolpert. If you’ve cited it somewhere, you might have used it to support the wrong conclusion. What it actually (vaguely) says is “You can’t learn from data without making assumptions”.The paper on the “No Free Lunch Theorem”, actually called "The Lack of A Priori Distinctions Between Learning Algorithms" is one of these papers that are often cited and rarely read, and I hear many people in the ML community refer to it when supporting the claim that “one model can’t be the best at everything” or “one model won’t always be better than another model”. The point of this post is to convince you that this is not what the paper or theorem says (at least not the one usually cited by Wolpert), and you should not cite this theorem in this context; and also that common versions cited of the "No Free Lunch" Theorem (continued...)
15 ways to read CSV file with pandas
pip install pandas. Next step is to load the package by running the following command. pd is an alias of pandas package. We will use it instead of full name "pandas". import pandas as pd
Moderna joins NumFOCUS Corporate Sponsors
The post Moderna joins NumFOCUS Corporate Sponsors appeared first on NumFOCUS.
Matplotlib Tutorial : Learn with Examples in 3 hours
MATLAB programming language and provides a similar MATLAB like interface for graphics. The beauty of this library is that it integrates well with pandas package which is used for data manipulation. With the combination of these two libraries, you can easily perform data wrangling along with visualization and get valuable insights out of data. Like ggplot2 library in R, matplotlib library is the grammar of graphics in Python and most used library for charts in Python. 
Write Short Blogposts
I encourage my colleagues to write blogposts more frequently. This is for a few reasons:
-
It informs your broader community what you’re up to, and allows that community to communicate back to you quickly.
You communicating to the community fosters a sense of collaboration, openness, and trust. You gain collaborators, build momentum behind your work, and curate a body of knowledge that early adopters can consume to become experts quickly.
Getting feedback from your community helps you to course-correct early in your work, and stops you from wasting time in inefficient courses of action.
You can only work for a long time without communicating if you are either entirely confident in what you’re doing, or reckless, or both.
-
It increases your visibility, and so is good for your career.
I have a great job. I find my work to be both
How to encourage participation in teleconferences
Participation is good!
How to drop one or multiple columns in Pandas Dataframe
- Drop or Keep rows and columns
- Aggregate data by one or more columns
- Sort or reorder data
- Merge or append multiple dataframes
- String Functions to handle text data
- DateTime Functions to handle date or time format columns
import command. import pandas as pd
import numpy as np
df = pd.DataFrame(np.random.randn(6, 4), columns=list('ABCD'))
A B C D
0 -1.236438 -1.656038
Google Renews Corporate Sponsorship of NumFOCUS
The post Google Renews Corporate Sponsorship of NumFOCUS appeared first on NumFOCUS.
NumFOCUS Hires First Ever Development Director
The post NumFOCUS Hires First Ever Development Director appeared first on NumFOCUS.
Labs update and May highlights
Time flies when you're having fun. Here is an update of some of the highlights of my second month at Quansight Labs.
The making of a black hole image & GitHub SponsorsBoth Travis and myself were invited by GitHub to attend GitHub Satellite in Berlin. The main reason was that Nat Friedman (GitHub CEO) decided to spend the first 20 minutes of his keynote to highlight the Event Horizon Telescope's black hole image and the open source software that made that imaging possible. This included the scientific Python very prominently - NumPy, Matplotlib, Python, Cython, SciPy, AstroPy and other projects were highlighted. At the same time, Nat introduced new GitHub features like "used by", a triaging role and new dependency graph features and illustrated how those worked for NumPy. These features will be very welcome news to maintainers of almost any project.
The single most visible feature introduced was GitHub Sponsors:
I really enjoyed meeting Devon Zuegel, Product Manager of the Open Source Economy Team at GitHub, in person after previously having had
SciPy and MathJax join NumFOCUS Sponsored Projects
The post SciPy and MathJax join NumFOCUS Sponsored Projects appeared first on NumFOCUS.
String Functions in Python with Examples
The cost of an open source contribution
Open source is massively successful. Some say it’s eating the world, although to my ears that phrasing doesn’t sound entirely like a good thing. Open source maintainers are always in need of help, and over the past years I’ve seen a lot of focus on ways open source projects can grow their communities and gain new contributors. Guidance on how to go about finding new contributors is easily found. E.TDK-Micronas partners with Quansight to sponsor Spyder
TDK-Micronas is sponsoring Spyder development efforts through Quansight Labs. This will enable the development of some features that have been requested by our users, as well as new features that will help TDK develop custom Spyder plugins in order to complement their Automatic Test Equipment (ATE’s) in the development of their Application Specific Integrated Circuits (ASIC’s).
At this point it may be useful to clarify the relationship the role of Quansight Labs in Spyder's development and the relationship with TDK. To quote Ralf Gommers (director of Quansight Labs):
"We're an R&D lab for open source development of core technologies around data science and scientific computing in Python. And focused on growing communities around those technologies. That's how I see it for Spyder as well: Quansight Labs enables developers to be employed to work on Spyder, and helps with connecting them to developers of other projects in similar situations. Labs should be an enabler to let the Spyder project, its community and individual developers grow. And Labs provides mechanisms to attract and coordinate funding. Of course
metadsl: A Framework for Domain Specific Languages in Python
metadsl: A Framework for Domain Specific Languages in Python¶Hello, my name is Saul Shanabrook and for the past year or so I have been at Quansight exploring the array computing ecosystem. This started with working on the xnd project, a set of low level primitives to help build cross platform NumPy-like APIs, and then started exploring Lenore Mullin's work on a mathematics of arrays. After spending quite a bit of time working on an integrated solution built on these concepts, I decided to step back to try to generalize and simplify the core concepts. The trickiest part was not actually compiling mathematical descriptions of array operations in Python, but figuring out how to make it useful to existing users. To do this, we need to meet users where they are at, which is with the APIs they are already familiar with, like numpy. The goal of metadsl is to make it easier to tackle parts
Why citations are not enough for open source software
A few weeks ago I wrote about why you should cite open source tools. Although I think citations important, though, there are major problems in relying on them alone to support open source work.
The biggest problem is that papers describing a software library can only give credit to the contributors at the time that the paper was written. The preferred citation for the SciPy library is “Eric Jones, Travis Oliphant, Pearu Peterson, et al”, 2001. The “et al” is not an abbreviation here, but a fixed shorthand for all other contributors. Needless to say many, many people have contributed to the SciPy library since 2001 (GitHub counts 716 contributors as of this writing), and they are unable to get credit within the academic system for those contributions. (As an aside, Google counts about 1,200 citations to SciPy, which is a breathtaking undercounting of its value and influence, and reinforces my earlier point: cite open source software! Definitely don't use this post as an excuse not to cite it!!!)
Not surprisingly, we have had
Spyder 4.0 takes a big step closer with the release of Beta 2!
This blogpost was originally published on the Quansight Labs website
It has been almost two months since I joined Quansight in April, to start working on Spyder maintenance and development. So far, it has been a very exciting and rewarding journey under the guidance of long time Spyder maintainer Carlos Córdoba. This is the first of a series of blog posts we will be writing to showcase updates on the development of Spyder, new planned features and news on the road to Spyder 4.0 and beyond.
First off, I would like to give a warm welcome to Edgar Margffoy, who recently joined Quansight and will be working with the Spyder team to take its development even further. Edgar has been a core Spyder developer for more than two years now, and we are very excited to have his (almost) full-time commitment to the project.
Spyder 4.0 Beta 2 released!Since August 2018, when the first beta of the 4.x series was released, the Spyder development team has been
(continued...)The Role of a Maintainer
What are the expectations and best practices for maintainers of open source software libraries? How can we do this better?
This post frames the discussion and then follows with best practices based on my personal experience and opinions. I make no claim that these are correct.
Let us Assume External ResponsibilityFirst, the most common answer to this question is the following:
- Q: What are expectations on OSS maintainers?
- A: Nothing at all. They’re volunteers.
However, let’s assume for a moment that these maintainers are paid to maintain the project some modest amount, like 10 hours a week.
How can they best spend this time?
What is a Maintainer?Next, let’s disambiguate the role of developer, reviewer, and maintainer
-
Developers fix bugs and create features. They write code and docs and generally are agents of change in a software project. There are often many more developers than reviewers or maintainers.
-
Reviewers are known
Two Sigma Renews Corporate Sponsorship of NumFOCUS
The post Two Sigma Renews Corporate Sponsorship of NumFOCUS appeared first on NumFOCUS.
Using GitHub for janky project reporting - some code
We scripted GitHub for lightweight project reporting
My first DNF (Ft Bragg 600k)
It's been six years since my first ride with The San Francisco Randonneurs and four years since my first 200k. I've ridden 18 rides that are at least that distance since then (3x 300k, 2x 400k, 1x 600k), completing my first Super Randonneur Series (2-, 3-, 4-, and 600k in one year) last year after not riding much the year before that. And this weekend I had my first DNF result on the Fort Bragg 600k. I Did Not Finish.
The best response to my choice of abandoning the ride to enjoy the campground came from Peter Curley, who said "That was a very mature decision." A clear departure from typical randonneuring stubbornness and refusal to give up, I celebrated my decision to quit as a victory when I arrived at the campground and made my announcement to the volunteers. I think I was so energetic about it that they did not believe me. I was being kind to myself, to my body, and at peace with the decision by
(continued...)Should I Resign from My Full Professor Job to Work Fulltime on Cocalc?
PyCon2019 poem
I'm back in Cleveland for another Pycon. Yesterday was my first full day here. Along with Matt Seale, I was a helper at Matthias Bussonnier tutorial ("IPython and Jupyter in Depth: High productivity, interactive Python). The sticky system is efficient at signaling when someone in a classroom needs help, and a lot of folks don't know that this practice was popularized by Software Carpentry workshops and continues to be used at The Carpentries.
I stepped out for a coffee refill and bumped into a large contingent of Bloomberg folks I'd never met (Princeton office). I guess we have something like 90 people at the conference this year, and I made the usual and true remark about how I go to conferences to meet the other people who work at our company. Then after his tutorial concluded, Matthias and I bumped into Tracy Teal, exchanged some stickers, and chatted about The Carpentries, Jupyter, organizing conferences, governance and sponsorship models, and a bunch of other stuff.
Matthias was a
(continued...)Why you should cite open source tools
Every now and then, a moment or a sentence in a conversation sticks out at you, and lodges itself in the back of your brain for months or even years. In this case, the sentence is a tweet, and I fear that the only way to dislodge it is to talk about it publicly.
Last year, I complained on Twitter that a very prominent paper that was getting lots of attention used scikit-image, but failed to cite our paper. (Or the papers corresponding to many other open source packages.) I continued that scientists developing open source software depend on these citations to continue their work. (More on this in another post...) One response was that surely the developers of the open source scientific Python stack were not scientists per se, and that citations were not a priority for them.
I still sigh internally when I think of it.
That tweet manifests a pervasive perception that open source scientific software is written by God-like figures. These massively experienced software developers have easy access to funds
Python Lambda Function with Examples
anonymous function as the function can be defined without its name. They are a part of functional programming style which focus on readability of code and avoids changing mutable data. 
lambda arguments: expressionLambda function can have more than one argument but expression cannot be more than 1. The expression is evaluated and returned. Example
addition = lambda x,y: x + yIn the above python code,
addition(2,3) returns 5
x,y are the arguments and x + y is the expression that gets evaluated and returned. Loops in Python explained with examples
Real World Examples of Loop
- Software of the ATM machine is in a loop to process transaction after transaction until you acknowledge that you have no more to do.
- Software program in a mobile device allows user to unlock the mobile with 5 password attempts. After that it resets mobile device.
- You put your favorite song on a repeat mode. It is also a loop.
- You want to run a particular analysis on each column of your data
Some questions and thoughts on journal peer review.
What's up with current peer review practice?
Create Dummy Data in Python
DataFrame functionimport pandas as pd
data = pd.DataFrame({"A" : ["John","Deep","Julia","Kate","Sandy"],
"MonthSales" : [25,30,35,40,45]})
A MonthSales
0 John 25
1 Deep 30
2
Things to think about when developing shotgun metagenome classifiers
Thoughts on goals and tradeoffs in classifying shotgun metagenome data.
Our submission for the NHGRI Human Genome Reference Center call
We wrote a big grant proposal!
PIP connection Error : SSL CERTIFICATE VERIFY FAILED
Error looks like this :
Could not fetch URL https://pypi.python.org/: connection error: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:598)
 |
| PIP SSL Certification Issue |
pip install --trusted-host pypi.org --trusted-host files.pythonhosted.org <package_name> -vvv
pip install --trusted-host pypi.org --trusted-host files.pythonhosted.org pandas -vvvInstall Python Package
Ways to Install Python Package
If Anaconda is installed on your system (laptop), click on Anaconda Prompt as shown in the image below. |
| Anaconda Prompt |
To install a python package or module, enter the code below in Anaconda Prompt -
pip install package-name
 |
| Install |
News from the NIH Data Commons Pilot Phase Consortium
The NIH Data Commons Pilot Phase Consortium is dead! (Long live the NIH Data Commons!)
Critically assessing open science - the CAOS meeting.
A summary of the CAOS open science meeting
Computational Psychiatry: Combining multiple levels of analysis to understand brain disorders - PhD thesis
I noticed that as my personal website at my former university went down that my PhD thesis could not be found anywhere, so I'm posting it here.
During my PhD I explored how machine learning and computational modeling of the brain can be used to improve our understanding, and diagnostics …
NIXML: nix + YAML for easy reproducible environments
The rise and fall of bioconda A year ago, I remember a conversation which went basically like this: Them: So, to distribute my package, what do you think I should use? Me: You should use bioconda. Them: OK, that’s interesting, but what about …? Me: No, you should use bioconda. Them: I will definitely look … Continue reading NIXML: nix + YAML for easy reproducible environments →Sustaining open source: thinking about communities of effort
Thinking about how to sustain open source.
My recent reading re sustaining open communities
What has Titus been reading lately?
Reduzindo a pilha
Sou fã de quadrinhos desde criança. As primeiras revistas que ganhei foram na primeira metade dos anos 90, alguns Mickeys, Mônicas, Trapalhões e X-Men. Em 98 comecei a comprar X-Men, Fabulosos X-Men e Wolverine, até os primeiros números da famigerada X-Men Premium. Sem dinheiro, enveredei pelos mangás e histórias fechadas. Quando a Panini começa a… Continue a ler »Reduzindo a pilhaThreat models for open online scientific engagement?
What threats are there for scientists in engaging in open online discussions?
Packaging PyQt5 apps with fbs — Distribute cross-platform GUI applications with the fman Build System
fbs is a cross-platform PyQt5 packaging system which supports building desktop applications for Windows, Mac and Linux (Ubuntu, Fedora and Arch). Built on top of PyInstaller it wraps some of the rough edges and defines a standard project structure which allows the build process to be entirely automated. The included …
Announcement: Audio TK 3.1.0
ATK is updated to 3.1.0 with heavy code refactoring. Old C++ standards are now dropped and it requires now a full C++17 compliant compiler. The main difference for filter support is that explicit SIMD filters using libsimdpp have been dropped while tr2::simd becomes standard and supported by gcc, clang and Visual Studio. Download link: ATK […]Telemulta
A voz feminina robótica (chegamos no tempo onde questão de gênero e robôs podem se confundir) soou, estranha e familiar como sempre, assim que o carro finalizou a curva para a direita: “Você entrou na Avenida Universitária; o limite de velocidade é 60 quilômetros por hora”. Meu pai sorriu e começou a falar: – Desde… Continue a ler »TelemultaMy foreword to "Bayesian Analysis with Python, 2nd Edition" by Osvaldo Martin
When Osvaldo asked me to write the foreword to his new book I felt honored, excited, and a bit scared, so naturally I accepted. What follows is my best attempt to convey what makes probabilistic programming so exciting to me. Osvaldo did a great job with the book, it is …
Call for Answers: Survey About Task Assignment
Professor Igor Steinmacher, from Northern Arizona University, is a proeminent researcher on several social dynamics in open source communities, like support of newcomers, gender bias, open sourcing proprietary software, and more. Some of his papers can de found in his website. Currently, Prof. Igor is inviting mentors from open source communities to answer a survey… Continue a ler »Call for Answers: Survey About Task AssignmentRevisiting authorship, and JOSS software publications
The question du jour: how should authorship on software papers be decided?
Using Bayesian Decision Making to Optimize Supply Chains
(c) 2019 Thomas Wiecki & Ravin Kumar
As advocates of Bayesian statistics in data science we often have to convince business-minded colleagues or customers of the added value of such an approach. While there are many good reasons for applying Bayesian modeling to solve business problems (Sean J Taylor recently had …
Mestrado em Ciência da Computação na UFPA 2019: Inteligência Computacional para Smart Grids; Metaheurísticas
Está aberto o processo seletivo para o mestrado em ciência da computação do PPGCC-UFPA. Nesse certame, estou disponibilizando 2 vagas para alunos que desenvolverão seus trabalhos junto aos demais pesquisadores no LAAI. As vagas são voltadas para os temas de inteligência computacional aplicada a Smart Grids e estudos sobre métodos metaheurísticos de otimização. Gostaria de… Continue a ler »Mestrado em Ciência da Computação na UFPA 2019: Inteligência Computacional para Smart Grids; MetaheurísticasLaKademy 2018
Em outubro de 2018, Florianópolis foi sede da sexta edição do LaKademy, o sprint latinoamericano do KDE. Esse momento é uma oportunidade para termos em um mesmo lugar vários desenvolvedores do KDE – tanto veteranos quanto novatos – de diferentes projetos para melhorarem os respectivos softwares em que trabalham e também planejar as ações de… Continue a ler »LaKademy 2018LaKademy 2018
Past October 2018, Florianópolis hosted the 6th edition of LaKademy, the Latin-American KDE sprint. That moment is an opportunity to put together several KDE developers – both veterans and newcomers – from different projects in order to work for improve their respective software and plan the promotional actions of the community in the subcontinent. In… Continue a ler »LaKademy 2018GPU Dask Arrays, first steps
The following code creates and manipulates 2 TB of randomly generated data.
import dask.array as da
rs = da.random.RandomState()
x = rs.normal(10, 1, size=(500000, 500000), chunks=(10000, 10000))
(x + 1)[::2, ::2].sum().compute(scheduler='threads')
On a single CPU, this computation takes two hours.
On an eight-GPU single-node system this computation takes nineteen seconds.
Combine Dask Array with CuPyActually this computation isn’t that impressive. It’s a simple workload, for which most of the time is spent creating and destroying random data. The computation and communication patterns are simple, reflecting the simplicity commonly found in data processing workloads.
What is impressive is that we were able to create a distributed parallel GPU array quickly by composing these three existing libraries:
-
CuPy provides a partial implementation of Numpy on the GPU.
-
Dask Array provides chunked algorithms on top of Numpy-like libraries like Numpy and CuPy.
This enables us to operate on more data than we could fit in memory by operating on that data in
Manage project dependencies with conda environments
TL;DR: Create a conda environment for each project, capture exact versions when possible, automate activation and updating with a bash function.
I often work on several different projects involving software: Python libraries, papers, presentations, posters, this website, etc. Each project has different dependencies and there is a non-zero chance that these dependencies might be in conflict with each other. For example, I need Python 2.7 to work on a tesseroid modeling paper with a student, while my current work on
(continued...)First Impressions of GPUs and PyData
I recently moved from Anaconda to NVIDIA within the RAPIDS team, which is building a PyData-friendly GPU-enabled data science stack. For my first week I explored some of the current challenges of working with GPUs in the PyData ecosystem. This post shares my first impressions and also outlines plans for near-term work.
First, lets start with the value proposition of GPUs, significant speed increases over traditional CPUs.
GPU PerformanceLike many PyData developers, I’m loosely aware that GPUs are sometimes fast, but don’t deal with them often enough to have strong feeling about them.
To get a more visceral feel for the performance differences, I logged into a GPU machine, opened up CuPy (a Numpy-like GPU library developed mostly by Chainer in Japan) and cuDF (a Pandas-like library in development at NVIDIA) and did a couple of small speed comparisons:
Compare Numpy and Cupy>>> import numpy, cupy
>>> x = numpy.random.random((10000, 10000))
>>> y = cupy.random.random((10000, 10000))
>>> %timeit bool((numpy.sin(x) ** 2 + numpy.cos(x) ** 2 == 1).all())
446 ms ± 53.1 ms perA quick read of _The genomic and proteomic landscape of the rumen microbiome_
Using short and long reads to assemble genomes from metagenomes!
Support Python 2 with Cython
SummaryMany popular Python packages are dropping support for Python 2 next month. This will be painful for several large institutions. Cython can provide a temporary fix by letting us compile a Python 3 codebase into something usable by Python 2 in many cases.
It’s not clear if we should do this, but it’s an interesting and little known feature of Cython.
Background: Dropping Python 2 Might be Harder than we ExpectMany major numeric Python packages are dropping support for Python 2 at the end of this year. This includes packages like Numpy, Pandas, and Scikit-Learn. Jupyter already dropped Python 2 earlier this year.
For most developers in the ecosystem this isn’t a problem. Most of our packages are Python-3 compatible and we’ve learned how to switch libraries. However, for larger companies or government organizations it’s often far harder to switch. The PyCon 2017 keynote by Lisa Guo and Hui Ding from Instagram gives a good look into why this can be challenging for large production codebases and also gives a good
(continued...)Anatomy of an OSS Institutional Visit
I recently visited the UK Meteorology Office, a moderately large organization that serves the weather and climate forecasting needs of the UK (and several other nations). I was there with other open source colleagues including Joe Hamman and Ryan May from open source projects like Dask, Xarray, JupyterHub, MetPy, Cartopy, and the broader Pangeo community.
This visit was like many other visits I’ve had over the years that are centered around showing open source tooling to large institutions, so I thought I’d write about it in hopes that it helps other people in this situation in the future.
My goals for these visits are the following:
- Teach the institution about software projects and approaches that may help them to have a more positive impact on the world
- Engage them in those software projects and hopefully spread around the maintenance and feature development burden a bit
We were invited by early adopters within the institution, both within the UK Met Office’s Informatics Lab
(continued...)The story behind the spacegraphcats project and paper
The TRUE story!